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Executive Summary 

Overview 

The Employment Training Panel contracted with the Management and Organization 
Development Center at California State University, Northridge to evaluate the role of 
structured on-site training (SOST) in its training program. Specifically, ETP wanted answers 
to these questions: 

1) Is the current delivery of SOST effective in ensuring that trainees are 
attaining competency in the skills being taught?” 

2) “Is the current SOST reimbm^ement rate appropriate?” 

3) ‘ “Are current SOST record keeping requirements appropriate/adequate?” 

4) What is the scope of SOST - in terms of the proportion of projects with 
SOST and ETP’s dollar investment in SOST? 

Methods 

We evaluated SOST from several perspectives, using the following research methods: 10 
case studies of active ETP projects with SOST, a survey of 50 recently closed projects with 
SOST, face-to-face interviews and focus groups with key stake holders, a Delphi survey of 
consultants, and a quantitative analysis of contract information. 

Results 

The results of our research can be summarized under the following topics: 

• Scope of SOST 

• Presence in projects 

We reviewed 1,066 contracts from FY 96-97 to present. We found that about 
half of these contracts included SOST funding. Slightly fewer than a third of the 
contracts (30.8%) had SOST scheduled in alljobs and about 15% had SOST 
present in some jobs. In the contracts we examined, the presence of SOST 
showed no trend over time. 

• ETP investment 

The Employment Training Panel spends about one of six dollars on SOST. For 
the 680 completed contracts, about 17% ($29.8 million) of the total training 
expenditures ($177.7 million) was spent on SOST. The disencumbrance rate for 
contracts with full SOST was 60% higher than for contracts with no SOST. 

• Value of SOST 

Our findings from this project support other published research which shows that 
SOST adds substantial value to ETP traimng. Trainees rated the quality and usefulness 
of SOST as significantly more useful than classroom or lab study and indicated that 
they thought SOST had a greater impact on productivity than classroom or lab 
training. In our survey of closed SOST projects, two thirds of managers rated SOST as 
equally or more effective than classroom training in meeting training goals. 
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• Strategic Issues 

Our research further identified six strategic issues related to the Employment Training 
Panel’s questions, as follows: 

1) SOST often reimburses existing training. Much SOST is on-demand 
troubleshooting as part of a supervisor’s regular duties; it does not supplement 
existing training efforts and therefore does not comply with ETP Legislation 
requiring ETP training to “supplement rather than displace funds available 
through existing programs conducted by employers and government funded 
training program . . .” 

2) Unstructured training practices are often used instead of SOST. Of the 10 active 
projects studied, only one had all five characteristics of quality SOST: a 
structured plan, trained trainers, on-site customized training, and a valid, reliable 
skill certification process. 

3) The system motivates contract administrators to focus on billing trainer hours 
rather than on achieving trainee competence. They spend more effort ensuring on- 
time completion and submission of paperwork than they do ensuring that trainees 
have met training goals. 

4) Reliable monitoring of SOST is difficult because much of it, as currently 
delivered, lacks the structure defined by the five characteristics of quality SOST. 

5) SOST reimbursement is often unrelated to actual cost. The median fully loaded 
hourly cost of deUvering SOST was $41 for companies and $101 for consultants. 
Both compames and consultants report a wide range of costs. 

6) At $80 per hour, SOST-only projects pay above the market price for one-on-one 
counseling or generic classroom training. 

Recommendations 

We present three policy alternatives for Panel consideration: 

• Maintain the status quo for the SOST portion of the reimbursement program; 

• Require that contractors have a plan to reinforce classroom training but only fund 
SOST training that meets new, rigorous structure criteria and, furthermore, stop 
funding SOST-only projects; or 

• Eliminate SOST fi'om the reimbursement program. 

We recommend the second alternative: 

• Require that contractors have a plan for reinforcing classroom training but only fund 
SOST training that meets new and rigorous criteria for structure. This will eliminate 
the reimbursement of trouble-shooting and informal coaching, documented as SOST, 
and will make SOST monitoring easier for ETP staff 

• Stop funding SOST-only projects. These projects do not fit the SOST model - they are 
just^a way to provide generic training to individuals or small groups who cannot be 
profitably served by classroom training. 

• Keep the existing $80 hourly trainer rate and continue to track trainer hours. 

Systematically evaluate the impact that any SOST policy changes might have on skill 
attainment, finances, or program administration. 
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Introduction 

In November 2001, the Employment Training Panel (ETP) contracted with the Management 
and Organization Development (MOD) Center of Caiifomia State University, Northridge to 
evaluate the role of Stmctured-On-Site-Training (SOST) in the ETP Program. The Panel 
asked that MOD focus on three questions: 

1. Does the current delivery of SOST ensure that trainees attain competence? 

2. Is the current SOST reimbursement rate appropriate? 

3. Are current SOST record-keeping requirements adequate? 

In this report, we begin with a discussion of stmctured SOST in general, then turn to what 
previous studies have shown about SOST as part of ETP, and will review the history of 
Panel policy on ETP. Next, we provide a brief overview of the methods employed in this 
study. This is followed by an analysis of the scope and cost of SOST training, and a section 
detailing the results of our evaluation by focusing on six strategic SOST-related issues that 
emerged from our work. Finally, we present a range of policy options and our 
recommendations. 

Research on the Effectiveness of SOST 

A great deal of research, from a variety of sources, shows clearly that learning on the job, 
compared to learning in the classroom, is a powerful training intervention. Hands-on 
training allows workers to leam new skills and knowledge in their work environment, 
leading directly to improved productivity, or helps workers to improve their performance by 
taking classroom learning and putting it into action at their workstation (Hart-Landsburger, 
Brauger, Reder and Cross, 1992 and Lave and Wenger, 1991). However, not all on-the-job 
or on-site training is equally effective. 

Since the guild system in the middle ages, various forms of on-the-job training (OJT) have 
been common. While OJT represents the most common form of training in American 
industry, its effectiveness is often suspect. An extensive study of unstructured OJT (Jacobs 
1994) concluded: 

"...while unstructured OJT occurs most often, employees seldom achieve 
the desired levels of expertise as a result of its use. . . .unstructured OJT 
leads to increased error rates, lower productivity and decreased training 
efficiency." 

As this quote points out, all OJT. is not the same. Much OJT is delivered in a haphazard way 
by co-workers and supervisors with limited resources and training expertise. At the same 
time, managers recognize that training on the job is powerfuL If classroom learning is to be 
transferred to the workplace, on-the-job reinforcement is crucial. The difficulty is creating 
OJT that works. Jacobs and Jones, in their recent book Structured On-the-Job Training: 
Unleashing Employee Expertise in the Workplace (1995), argue convincingly that the key 
ingredient missing from most OJT is "structure.” The authors’ definitions of unstructured 
and structured OJT are given in Figure 1 on the next page. We find these definitions 
valuable for evaluating ETP's delivery of SOST. 
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Figure 1 







Employee is trained by an experienced 
employee whose experience as a trainer is 
likely to be limited and whose task expertise 
may also be questionable. Training content, 
methods and outcomes vary across 
employees. 


Employee is trained by an experienced 
employee who has expertise as a trainer and 
in the task to be learned. Training content, 
methods, and outcomes are consistent 
across employees. 



Our own research on ETP's SOST suggests that the authors are correct. On-the-job training 
is a potentially powerful intervention, but must be carefully structured to be fully effective. 

Previous Research on EXP SOST 



our recent study, ETP at Work, we conducted case studies of 26 ETP projects, most 
including a SOST component. Through interviews with project managers, trainers and 
trainees, and an evaluation survey of trainees at each site, we assessed SOST effectiveness. 

We found that SOST quality varied widely across the projects we visited. SOST is a 
powerful instructional approach when applied correctly but, all too often, we found that 
SOST activities had limited training relevance, were poorly supervised, and contributed 
httle to training effectiveness. 

It is important to note that the projects we studied operated under a different SOST policy 
than what is in operation today. At that time, pajmients were based on trainee hours spent 
on SOST assignments. The policy allowed 10 trainees per instructor for SOST training. 
Each trainee had to document every hour spent on SOST training. The current policy pays 
for instructor time spent on SOST and requires documentation only for instructor time. The 
focus is now on measuring trainee achievement of competence in specified skills. Thus, one 
could reasonably assume that practices, and possibly the impact of SOST, have changed 
since our fieldwork. 

Our research found that good SOST has a number of key characteristics. First, the 
assignments follow, in a timely manner, the topics covered in class. Second, the assigned 
tasks deal with immediate, work-related problems. Finally, trainees need attention fi-om 
instmctors while they complete SOST assignments - employees need to be able to get help 
promptly when they “hit a road block” in their assignment, and they need to get immediate 
feedback on the quality of their work to know whether they are using new skills effectively. 

In our fieldwork, we found that SOST was often poorly delivered. This observation was 
confirmed by our questionnaire - across all projects, trainees rated the quality of SOST 
substantially below overall training quality (SOST was rated 2.89 and overall training 3.14 
as illustrated in Figure 2 on the following page). It appears that, in some cases, SOST was 
added to contracts to increase their value without increasing the hours employees were off 
the job, and so, it seems, was half-heartedly implemented. 
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Figure 2 



Trainee Rating of Training Quaiity 



Quality of Training Overall 
Time on Each Topic 
Effectiveness of SOST 
Degree Training Customized to Co. 

Right Level for Trainee 
Usefulness of Topics 
Ability of Trainers to Hold Interest 
Quality of Instructional Materials 
Clear Objectives 
Quality of Instructors 




■ Average Rating 1=Poor, 2=Fair, 3=Good, 4=Excellent 



Trainees often complained that SOST seemed a waste of time. In interviews about SOST, 
the topics that came up repeatedly were the difficulty of completing SOST documentation 
and the pressure that employees felt to complete assignments. Consequently, when asked to 
recall their SOST experience, learning was seldom the first thing that came to mind. It may 
be that the newer SOST policy has mitigated these problems. The text box on the following 
page illustrates an effective SOST program. Again, it is important to note that these 
observations refer to a time when ETP was tracking employee hours, not just instructor 
hours. 



Figure 3 



SOST at Basic Batteries: The Right Way to Do SOST 

In our questionnaires, 56.3% of the trainees at this site rated the effectiveness of SOST 
as “good” and 31.3% thought it was “excellent.” SOST worked well here because the 
company used in-house trainers who knew the production processes; the trainer was 
the union shop steward so the company had the cooperation of the union; and the 
supervisors were extremely cooperative in helping trainees as they completed their 
SOST assignments. One trainee we interviewed felt encouraged to use the new skills 
and knowledge due to “the on-site training that we received.” 



To explore the relationship between training quality and training impact, we used regression 
analysis. Regression is a statistical technique that measures the strength of the relationship 
between predictor variables and a target variable such as "impact of training." As shown in 
Table 1, we used regression to see whether different aspects of the training quality were 
associated with training impact. The training impact was measured three ways: the amount 
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learned, how often new skills were used, and productivity change. The coefficients 
measure the strength of the umque relationship between a quality measure and an 
outcome measure. Significance measures show the probability that the measured 
relationship is due to chance or random error. Traditionally, relationships are considered 
statistically sigmficant only at levels of 0.05 or lower (indicating a five percent or lower 
probability that the relationship is due to chance or random error). Significant 
relationships are shown in bold t}'pe. Finally, the overall measure indicates what 
proportion of variation in impact measure is accounted for by the measures of quality, in 
aggregate. In other words, tMs measure shows the degree to which quality measures 
predict training impact. A higher that the apparent correlation between the predictor 
quality measures and the observed training impact is stronger. 



Table 1 









'&^S-.;^&SkiljiiUJs'edfe,-J^^ 






^Slgnlficaj^ 










Clear objectives 


.048 


.481 


.028 


.701 


.044 


.527 


Usefulness of topics 


.159 


.013 


.311 


.000 


.249 


.000 


Length of time on 
topics 


-.034 


.620 


-.059 


.409 


.041 


.566 


Quality of 

instructional 

materials 


.065 


.315 


.067 


.316 


.001 


.983 


Degree of 
customization to 
company 


.127 


.043 


-.060 


.356 


.086 


.166 


Quality of instructors 


-.145 


.056 


-.124 


.126 


-.028 


.712 


Effectiveness of 
SOST 


.306 


.000 


.344 


.000 


.258 


.000 


Ability to hold 
interest 


.164 


.032 


.039 


.624 


.082 


.283 


Right level 


-.037 


.554 


-.055 


.391 


-.089 


161 


Quality Overall 


.040 


.612 


.101 


.220 


.093 


.240 


Adjusted 


.284 




.310 




.343 





The table shows that, while we observed extensive variation in SOST quality, there was a 
strong relationship between SOST quality and training impact. In other words, as SOST 
quality increased, its impact - on amount learned, fi-equency of skill use, and productivity 
- all increased significantly. This led to our conclusion that SOST remains an important, 
if imperfect, part of the ETP program. 

A Brief Policy History of ETP 

SOST has been apart of ETP since inception. Previous panels recognized that 
opportunity to build skills on the job was a critical element of training. Since ETP's early 
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years, the management of the SOST component has been problematic. Panels dealt with 
SOST issues as they emerged. Below, we provide a summary of major recent SOST 
issues and describe the policy responses. 



Figure 4 













1995 


Some projects included a large number 
of SOST hours but the value added was 
not always clear. 


Panel restricts SOST to a ratio of two 
SOST hours for each class hour. 


1996 


ETP requires tracking of every hour of 
SOST completed by every trainee. This 
is found burdensome for employers. In 
addition, it is clear that some trainees 
need more SOST than others. 


Panel changes from tracking trainee 
hours to tracking instructor hours. 

Compensation changes from $8 per 
trainee hour to $80 per instructor hour. 
Payment is based on a 10 to 1 trainee to 
trainer ratio in either case. 

The SOST instructor must still certify 
every trainee’s skill attainment. 


1999 


The power of SOST is recognized as 
particularly appropriate for Welfare-to- 
Work participants. New policy of 
tracking instructor hours allows for one- 
on-one training. 


Panel experiments with Welfare-to-Work 
training, including only SOST and 
allowing one-on-one training. 



SOST In Other States 

We reviewed two national studies on state- financed training programs and found that 
only four states - Illinois, New Jersey, Pennsylvania and Missouri - have programs that 
fund on-the-job training similar to SOST. In New Jersey, on-the-job training is defined as 
“training conducted at the work station in which limited production is generated by the 
trainee as an outcome of the training process” (Regional Technology Strategies, Inc. 

1999, p.64). 
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Methods 

Our approach to the questions posed by ETP was a method known as triangulation. In 
this approach, researchers measure a phenomenon from multiple perspectives. ETP is a 
complex system with an independent panel, a central administration, regional offices, and 
a wide range of contractors and subcontractors. Each element of the system is sure to 
provide a different perspective, and the reality is likely to lie somewhere between the 
various stakeholders’ perspectives. To answer the research questions, we designed a 
method that draws on the perspectives of all ETP participants, adds the best available 
empirical data, and uses the knowledge we have accrued in fifteen years of studying ETP. 

In short, we combined the following methods to answer the questions: 

• Interviews with ETP executives and Panel members 

• A focus group with SOST monitors 

• Case studies of ten current representative ETP projects with SOST 

• Interviews with internal project managers, trainers and consultants 
. A survey of SOST Trainees 

• A telephone survey of managers of recently closed SOST projects 

• A Delphi Survey of contractors and consultants 

• An analysis of five years of contract data 

Table 2 on page 9 shows how method components tie to the research questions posed in 
the RFP. A double star indicates a primary source for answering the question posed; a 
single star indicates a secondary source for answering that question. 
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A synopsis of research methods is given below. Samples of actual instruments used to 
collect data are included in Appendices A through I on pages 55-75. 

Interviews of ETP Organization Members 

The first step was to interview representative key stakeholders within the ETP 
organization. 

• Interviews With Selected Panel Members 

Three Panel members were designated by the acting executive director to be 
interviewed. The Panel members were questioned on the following topics: current 
SOST issues, ideal SOST components versus recent problems, the role of SOST in 
the delivery of ETP projects, evaluation of reimbursement rates, and SOST policy 
evolution over the past several years. The purpose of these questions was to 
increase understanding of overall issues and to collect key policymakers’ views on 
the current state of SOST. 

• Interviews With ETP Executives 

Interviews with ETP executives focused on the context of SOST policy. Specific 
topics included: recent policy history of SOST, emerging issues that led to this 
study, and the role of central and regional office staff in implementation. We also 
collected executives’ views on the value and effectiveness of SOST. 

Focus Group With Monitors 

Because monitors are on the fi-ontline of ETP, spending time with contractors and 
consultants every day, we wanted to collect their views on key project questions. We 
also solicited their views on how best to observe SOST in the field. This exercise helped 
sharpen the key-issue focus for field site visits. 

Ten Site Visits 

This component was at the heart of the research. Our previous study showed us the value 
of getting into the field to observe training on site. We believe this is the key to 
answering the questions posed. In effect, we treated each site visit as an individual case 
study. We developed a project history by reading the contract file, interviewing the 
contract monitor, and then going into the field to see the project at work and talk to 
managers, trainers, and trainees. The key elements of this approach are reviewed below. 

• Sample 

We worked with ETP staff to identify thirty companies with current SOST projects 
representing the range of projects funded by ETP. Ten were selected for study and 
20 as potential replacements. In the selection, we considered variables such as 
region, industry, size, and training provider. However, we had to make imm ediate 
substitutions because not all compames originally chosen would agree to 
participate. We actually contacted twenty-one companies to get ten case-study 
participants. Some companies declined for security reasons. Others said they were 
too busy to participate. At a few, we were unable to make contact with the 
appropriate training person. We pulled the contract files on those who agreed to 
help us and extracted relevant data. 
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• Interviews with Internal Project Managers and Trainers 

Through interviews with companies’ SOST project managers and trainers we found 
out why the company or trainer became involved with the ETP program, why a 
SOST component was included in the project, and how the pl annin g and 
implementation of SOST unfolded. We also collected data on actual costs of 
implementing SOST and found out how ETP reimbursement was spent. Please see 
the data collection instruments in Appendices C and G on pages 59-60 and 70-71, 
respectively. 

o Comsmltamt Imterviews 

We interviewed the contractor’s consultant, if any. These interviews addressed the 
goals, benefits, costs (reimbursements), implementation procedures, and 
effectiveness of SOST. Consultants were also asked whether or not they 
participated in the decision to include SOST in the project, and what they would 
have done differently had ETP support not been available. We discussed the 
consultant's interaction with the project monitor and evaluated consultant costs as 
part of ETP reimbursement. 

• Survey of SOST Trainees 

We asked current SOST trainees to rate, on a structured evaluation form, the 
quality, effectiveness, and impact of the SOST they received. The form was based 
on our earlier experience in administering evaluation instruments at ETP sites. 

• Observing the process 

Researchers shadowed a SOST instructor to observe time use, content of SOST 
assignments, number of participants contacted in an hour, types of one-on-one 
instruction, and any visible impact of the instruction (such as trainee use of the 
techniques taught). 

Survey of Fifty Closed Projects 

Fifty companies with recent SOST were contacted by telephone. The projects were 
chosen from a list of 200 provided by ETP’s Sacramento office. The most recently 
completed projects were contacted first, but all 200 companies were contacted eventually 
in order to get fifty responses. 

Based on our experience with case studies, we developed a survey instrument for project 
managers of the fifty SOST projects that closed most recently. The purpose of the survey 
was to collect data on manager experience relevant to each research question. We 
wanted to validate our tentative findings from the ten case studies. For example, 
knowing the cost factors identified by case studies, we could ask managers for cost data 
on these factors. This provided a database for estimating cost across all projects. We also 
asked managers for their experiences with the administrative and recording-keeping 
aspects of the project. 

Survey of Data with Delphi Method 

We designed and applied a modified Delphi survey of consultants and contractors, to 
explore their perceptions of issues relevant to the three research questions. The Delphi 
techmque, developed nearly fifty years ago, is a form of iterated survey analysis. A wide 
range of acknowledged experts in the subject area are identified and this initial group is 
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surveyed on the research topic. We acquired a sample of forty-three e-mail addresses, 
from the Alliance for ETP and from a search of consultant Web sites and various lists of 
ETP contractors. Of forty-three initially contacted, sixteen participated. Our Delphi 
sixrvey involved three phases: 

• Phase I 

The purpose of the first phase was to obtain basic information about cohort 
perceptions and to find out if perceptions of the effectiveness of SOST differed 
depending on how SOST training was provided. In this initial query, each 
respondent was asked to complete our survey and, if the respondent had special 
expertise or a strong opinion, to write an explanation of the answers given. We 
collected these responses and assembled them into a composite response sheet that 
preserved respondent anonymity. This sheet provided representative answers and 
the percentage of sample that seemed in agreement each answer. The composite 
answer sheet, the compilation of explanations, and a fresh copy of the original 
survey form were then distributed to each respondent. 

• Phase II 

In the second phase, the panel was asked to review the compiled answers and 
explanations and to provide a second set of answers to the same questions. Survey 
participants who had strong opinions regarding an answer were again invited to 
write explanations. 

• Phase III 

In the third phase, the respondents were given statistical summaries of previously 
answered questions, and a series of representative statements gathered from 
comments, to which they could express the strength their agreement. Though the 
study was designed to allow the possibility of a fourth or even fifth round, sufficient 
convergence of answers was achieved in three iterations. 

In order to minimize the cost of participation, to maximize response rate, and to facilitate 
quick tum-around, we collected data by e-mail. We contacted participants by telephone, 
letter, or e-mail and explained the modified Delphi process. We distributed the 
questionnaires, answer summaries, and explanation compilations electronically using a 
secure Web site. To remind respondents to participate, we set frequent e-mails with links 
to the Web site. 

We used two different survey approaches to estimate the actual cost of SOST training.To 
obtain companies’ estimates of the cost of in-house SOST training, we included cost 
questions in the telephone survey of companies with recently closed projects. To obtain 
consultants’ estimates of their cost of providing SOST training, we included cost 
questions in the modified Delphi survey. These two surveys provided interesting results 
(summarized in Figures 14, 15, and 16 on pages 38-40) regarding the actual costs of 
providing SOST training. 
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Data Analysis 

In accordance with our triangulation approach, we analyzed data from several sources to 
answer each research question. Following is a brief description of how data was used to 
answer the research questions. 

1. Does the current delivery of SOST effectively ensure trainee attainment of 
competence in the skills being taught? 

We answered this question by first observing the delivery of SOST training in the 
field and then surveying current SOST trainees to get their perceptions of the 
quality and value of the training. Interviews or surveys of all other key stakeholders 
— ETP staff, monitors, trainers, and 50 recent project managers — completed the 
research. The combined data from all sources enabled us to draw conclusions about 
SOST delivery and effectiveness. 

2. Is the current SOST reimbursement rate appropriate? 

During the case-study fieldwork, we identified the factors driving overall SOST 
cost and collected available on-site comparison data, such as costs of other on-the- 
job training. We collected additional cost data in the Delphi survey of trainers and 
in the survey of managers of 50 closed projects. Data from all sources were 
combined to provide a complete picture of the costs of SOST. This picture was 
used to evaluate current reimbursement policies and generate recommendations for 
reaching Panel policymakers’ goals for SOST. An essential part of this analysis 
was outlining the incentives that various fee structures create for contractors and 
consultants. 

3. Are current SOST record-keeping requirements adequate? 

Here too, the question was answered by first making objective observations of 
record-keeping in the field and then collecting the views of parties directly 
involved. Data on these parties’ perceptions were combined with our own 
observations and conclusions drawn about the most effective method of 
documenting SOST. 
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Results 

In the process of piecing together the data to answer the research questions, we were led 
to the conclusion that there was an important question not initially posed. What is the 
extent of SOST, in terms of projects involved and money expended? The results section 
begins by showing the size and scope of ETP's investment in SOST and then describing 
the impact of the SOST we studied. We then discuss six key strategic issues that 
emerged from our work: 

1 . SOST funds often reimburse existing training. 

• 2. Unstructured training practices are used instead of SOST. 

3. Contract administrators focus on billing trainer hours rather than on achieving 
competence. 

4. Reliable monitoring of SOST is difficult. 

5. SOST reimbursement is often unrelated to actual costs. 

6. SOST-only projects pay more than market price for one-on-one counseling and 
generic classroom training. 

In our view, some of these issues represent forces hindering SOST in reaching its full 
potential for trainee skill attainment and productivity improvement. Others are strategic 
policy matters, related either to price or to ability to monitor activities, which the Panel 
may wish to consider as they review SOST. 

Scope of SOST 

It is impossible to review SOST without first understanding the extent of ETP investment 
in this training method. Overall, ETP spends one dollar out of six on SOST. ETP 
budgeted $15.2 million for SOST in Fiscal Year 2000-01. To understand the extent of 
SOST in some detail, we analyzed 1,066 ETP contracts to look for patterns and trends in 
SOST funding. This analysis answered the following questions: 

• What percentage of ETP contracts includes SOST funding? 

• What percentage of ETP training dollars goes to supporting SOST? 

• Are there time trends in the number of contracts with SOST funding or in the 
percentage of training dollars devoted to SOST funding? 

• Is there evidence that the presence of SOST funding might compromise the 
success of a training contract? 

• Is there a relationship between the use of consultants in the contract and inclusion 
of SOST funding? 

Our analysis used data from 680 completed contracts and 396 active contracts, for a total 
of 1,066 contracts. The completed contracts included those that began in third quarter 
1996 and completed before December 2001; the active contracts were those active in 
January 2002, when field visits were scheduled. 
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The Prevalence of SOST in contracts 

Nearly 50% of contracts studied contained SOST funding, and there was no significant 
time trend in this percentage. The percentage of contracts with SOST funding averaged 
46.3% across all completed and active contracts - it was 44.9% of completed contracts 
and 49.7% of active contracts. The slightly higher percentage of active contracts with 
SOST funding was offset by the fact that the SOST in active contracts involved fewer 
jobs. 

Contracts with SOST may have SOST for all jobs' or only for some jobs. Overall, 30.8% 
of contracts had SOST for all jobs (Full-SOST) and 15.2% had SOST for only some jobs 
(Partial-SOST). The ratio of Full-SOST to Partial-SOST is 2 to 1. Completed contracts 
had a higher ratio of Full-SOST (34.3%) to Partial-SOST (10.6%) than did active 
contracts, which were split almost evenly between Full-SOST (25.3%) and Partial-SOST 
(24.5%). Figure 5 on the next page shows the percentage of contracts with SOST in 
some or all jobs, by the quarter in which the contract originated. Clearly, the percentage 
of contracts with SOST funding is highly variable from quarter to quarter, and the 
analysis revealed no significant time trend in the presence of SOST in contracts overall - 
for example, 50.5% of contracts included SOST in 1997 compared to 51.6% in 2001. 
However, there is a clear decrease in the percentage of Full-SOST contracts over time, 
moving from an average of 41.5% Full-SOST contracts in 1997 to only 26.0% Full- 
SOST in 2001. 



’ A “job” is defined as the training of a specific group of workers in a specific skill. For example, 
management staff might get computer-skills training, assembly workers might get continuous 
improvement skills training, and lab workers might get manufacturing skills training. Some contracts 
have only one job; others have as many as 34 - the average number of jobs per contract is just under 6. 
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The Level of SOST Funding 

The level of SOST funding in contracts varies from year to year, but we found no 
significant trends over time. Figure 6 on page 18 shows the total planned training 
expenditure for contracts in each of the last five fiscal years, along with planned SOST 
expenditure in those contracts. While planned SOST expenditure varied from 12% to 
24% of total, there is no apparent time trend in the SOST percentage. Over the full five- 
year period, planned SOST expenditure averaged 18.2% of total planned training 
expenditure. 

Actual training expenditures in contracts are usually less than planned because of 
disencumbrance of funds, which is caused by non-completion of some of the planned 
training. Reasons for non-completion vary - fewer workers than planned may start the 
training, some workers may quit or be transferred before the end of training, or some 
change in market conditions may lead management to cancel part of the planned training. 
Whatever the cause, the actual expenditure for overall training and for the SOST 
component are usually less than planned. An analysis of the 680 completed contracts 
revealed that, of the $ 1 77,725,083 spent on the contracts, SOST accounted for 
$29,825,328, or 16.8% of the actual training expenditures. The absence of any time trend 
in the SOST percentage indicates that, under current policies, ETP can expect to continue 
spending one of six dollars on SOST. 
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Other SOST Patterns - Disencumbrances and Consultants 

The analysis of completed contracts also disclosed an interesting relationship between 
SOST inclusion and the disencumbrance rate. When the 680 completed contracts were 
grouped into No-SOST, Partial-SOST, and Full-SOST, the average disencumbrance rate 
varied directly with the level of SOST in the contracts. Table 3 on page 20 shows that 
Full-SOST contracts disencumbered 47.4% of their original funds; Partial-SOST 
contracts disencumbered 31.4% of original funds; and No-SOST contracts 
disencumbered only 28.9%. The disencumbrance rate for Full-SOST contracts is 60% 
higher than that for No-SOST contracts. 

Grouping the completed contracts also revealed a strong positive correlation between 
consultant involvement and SOST inclusion. Consultants were involved in 64.4% of 
Full-SOST contracts, 54.2% of Partial-SOST contracts, and only 36.8% of No-SOST 
contracts. The reasons for this pattern are unclear. In our interviews with staff, we found 
that some believed that consultants encouraged contractors to “load up” contracts with 
SOST, because it was “easy money.” However, the higher disencumbrance rate for 
projects with more SOST suggests that SOST money may not be easy to earn. 



The Role of Structured-On-Site-Training In ETP - Final Report 



19 



Management and Organization Development Center. California State University, Northridge 




fS 



o 

Ck 

"3 

a 



Oh 

H 

a 

NN 

a 

a 

A 

H 

•mm 

CO 

I 

a 

O 

In 

:3 

CO 

o 

o 

0 ) 

ja 

H 



O 

ERIC 



O P 
<^0 



BEST COPY AVAILABEiS 



Management and Organization Development Center. Caiifornia State University, Northridge 



The Value of SOST 

Our field observations support published research (reviewed earlier), that SOST adds 
substantial value to training. It remains a powerful method for ensuring that skills are 
learned and that learmng is put to work on the job. We observed that more structured 
training results in better skill attainment. For example, AA Electro had the most 
structured training plan of any of the ten case-study sites visited. The assembly of each 
product was broken down into a series of tasks to be mastered by the trainee. Production 
assemblers cannot assemble any of the company’s products without passing a fully 
documented certification process that requires the trainee to assemble a product task by 
task until it is complete and the quality of the finished product is thoroughly tested. This 
is done under the scrutiny of a trainer who evaluates the trainee at each step of the 
process. The trainer can objectively verify that the trainee has acquired the necessary 
skills, because he has actually watched the skill in action as the product is assembled. 
There is no ambiguity — the product either has or has not been assembled correctly. The 
regimentation in the training process at AA Electro ensures that all trainees are 
consistently trained and evaluated. 

Figure 7 

Usefulness of SOST vs Classroom Instruction 




Mean Rating on Scale where 1=Not Useful at All and 10=Very Useful 
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Figure 8 

SOST vs Classroom Impact on Productivity 




Mean Rating on Scale where 1=No Impact and 10=Positive Impact 



Trainees in our survey gave about equal ratings to overall classroom or lab quality vs. 
SOST quality, and to classroom or lab instructor quality vs. SOST instructor quality. 
However, Figure 7 shows that trainees rated SOST usefulness significantly higher than 
classroom or lab usefulness - 8.2 for SOST and 7.9 for classroom or lab (on a ten point 
scale where 1 = “Not Useful at All” and 10 = “Very Useful”). In addition. Figure 8 
shows that trainees across all ten case-study sites rated the productivity impact of SOST 
slightly higher than classroom productivity impact - 7.8 compared to 7.6 (on a ten-point 
scale where 1 = “No Impact” and 10 = “Positive Impact”). 

In our recent survey of fifty closed SOST projects, about two thirds of managers (65.9%) 
indicated that SOST is as effective as, or more effective than, classroom training in 
achieving their training goals. When we asked the managers to estimate how much of the 
added value they had expected to come from classroom training compared to SOST, they 
said they had expected about half of the added value to be derived from classroom and 
half from SOST. They then confirmed that the actual contribution of value added was in 
fact half and half. 

Six Key Strategic Issues: 

Issue #1: SOST Funds Often Reimburse Existing Training 

At one manufacturing site, during a fifteen-minute interview with a trainer employed as a 
full-time supervisor, the walkie-talkie strapped to his shoulder buzzed four times. Each 
calls was a “problem” that had occurred on the shop floor. The problems covered a range 
Oi issues — one call was about a disciplinary action that was neededj another was an 
assembly process that was not going smoothly. The latter type of call could result in 
booking of SOST time. 

This on-demand troubleshooting, while acceptable as SOST according to the site’s ETP 
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contract, is a routine part of the supervisor’s duties. At the same site, one supervisor said 
that 90% of the training he performed (reimbursed as SOST) was of this type. On- 
demand troubleshooting may be excellent management, but in these instances no new 
training is created by the SOST funding. In other words, at some locations SOST dollars 
are subsidizing business as usual. 

ETP Legislation requires that ETP training should "supplement rather than displace, 
funds available through existing programs conducted by employers and government 
funded training programs. In situations where supervisors provide SOST as part of 
their regular duties, it is easy to argue that ETP funding is not "supplementing" existing 
training efforts. 

Of administrators of closed SOST projects, 36% reported that a primary goal of SOST 
was to provide on-demand help with problems when trainees returned to their jobs; 38% 
reported that this on-demand tutoring in response to problems actually did occur. On the 
whole, our research suggests that roughly one-third of training reimbursed as SOST could 
be classified as traimng that the company would have done even if no SOST 
reimbursement had been available. 

In the survey of 50 administrators of closed SOST contracts, 25% of respondents said 
that — absent the ETP SOST program — the company would have done exactly the same 
amount of OJT and would have footed the bill with company funds. Another 47% 
indicated that the company would have done some of the same OJT, but not as much. 
These figures indicate that a minimum of 25%, and an absolute maximum of 72% (25% + 
47%), of all ETP dollars paid to reimburse SOST goes to subsidize routine tr ainin g. 

Three additional facts support the conclusion that SOST dollars often subsidize business 
as usual. First, information gathered in interviews during field visits make it clear that, at 
six of the ten sites visited, the same OJT would have occurred without SOST funding. 
Second, only 36% of the administrators of closed SOST projects responded “Yes” when 
asked if SOST caused any lost production. Logically, non-routine SOST should cause 
lost production. If trainees are completing planned exercises using new skills, tools, or 
materials, it is logical that production will lag. Third, the dispersal of SOST funds in the 
company shows that SOST does not add costs — it is viewed as part of routine 
supervision. At one site, the training manager reported he "used the SOST funds to 
refurbish the training room." At another site, SOST funds were used to pay for diversity 
training because funds were not available for this training in the regular budget. In the 
view of the training managers, SOST money was available for other purposes because 
there was no marginal cost for delivering SOST, since it was delivered by supervisors as 
part of their regular duties. In other cases, contract administrators interviewed during 
field visits reported that SOST reimbursements were captured by the “finance people” in 
the company, rather than being accessible to training decision-makers. This finding may 
be partly because of quirks in the managerial accounting systems at those particular 
companies, but the underlying message is that funds are going elsewhere because no new 
costs were generated by SOST. 

One reason compames report that SOST funding is not creating new training at their sites 
is that, at many companies, the training identified in the SOST contract was cited as 

^ Employment Training Panel Legislation 10200 (a) 4. 
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something that the company must do in order to be a viable competitor. In other words, if 
the company didn’t do OJT (funded as SOST), then the company might as well close its 
doors. Had SOST funding not been available, would the company have paid for the 
training out of company funds, cut back operations, or shut the operation down entirely? 

It is important to note that the examples and evidence cited in this section are in no way 
intended as exemplary of poor management or poor supervision. In fact, the supervision 
we observed during field visits was fi'equently superb. However, superb supervision and 
trouble-shooting do not necessarily merit SOST funding. This is something of a dilemma 
for poUcymakers. On one hand, most SOST dollars do appear to fund - though not 
always, create - value-added activities that increase competitiveness. On the other hand, 
many value-added activities that support competitiveness are simply business as usual, 
and funding that supports these activities could be viewed as a subsidy. We interviewed 
some stakeholders who were not troubled by this because they viewed the availability of 
a SOST subsidy, regardless of how it was used, as an incentive for companies to remain 
and expand in California. 

Issue #2: Use of Unstructured Practices Instead of SOST 

Our field observations, the survey of closed projects, and interviews with ETP staff all 
confirm that, in many projects, informal coaching substitutes for carefully structured 
SOST. As noted, based on our experience and previous research, the five characteristics 
of SOST that can ensure skill attainment are: 

A Structured Plan: 

Plan includes sequenced modules, measurable training objectives, and specific on- 
the-job activities tied to class instruction. 

Trained Trainers: 

Trainers are trained in both content and instructional methods. 

^ Customization to Companies: 

Training uses company's products, tools, machines, and procedures. 

^ Delivery On Site: 

Training is done at the workplace and on the clock. 

^ Valid, Reliable Skills Certification: 

There is objective, fi-equent measurement of skill attainment tied to job 
performance. 

These characteristics were often missing fi-om the field sites we visited. Table 4 on page 
32 shows that only one of the sites visited had all of the characteristics of effectively- 
structured SOST. Furthermore, only five of the ten sites had a structured training plan, 
six had trained trainers, seven were fully customized, eight were delivered on site, and 
only three had valid skills certification. 

In this analysis, we review each of the five characteristics of quality SOST and what we 
learned about its presence or absence. 

o A Structured Plan 

ETP requires that ETP contracts including SOST must have a plan. We examined a 
number of contracts and found that the plans often lacked the specificity needed to 
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produce structured training: They did not have specific training modules with 
objectives, or measurable training objectives. The sample below is an example of 
the type of plan often approved. 



Figure 9 



■/ 



SOST Training Activity Plan for Continuous Improvement 
Coach Trainees in using continuous improvement techniques 
Coach Trainees in identifying process problems 
Coach Trainees in improving customer satisfaction 
Observe/ Coach Trainees on team communications 
Observe/ Coach Trainees on operating procedures 
Observe/Coach Trainees how to satisfy customer needs 



SOST Training Activity Plan for Computer Skills 
Review Computer Systems with Trainees 

Assist Trainees in retrieving data needed during a typical work 
project 

Observe trainees creating spreadsheets/ narrative documents and 
navigating through multiple software applications 

Review data inputs and assignments 



This plan obviously calls for unstructured coaching, not structured training. When 
we analyzed our ten case-study sites, we found four of them had no structured plan 
for SOST, one had a partially structured plan, and five had fully structured plans. 
Tech-Loan^ is an example of a company without a structured SOST plan. Tech- 
Loan is a financial services company in the Central Valley. Loan officers were 
trained in operating sophisticated computer systems that supported the loan 
application process. The SOST process we observed consisted of loan officers 
summoning supervisors for help when they were having problems with the 
software. This was logged on SOST time sheets as coaching related to training. 

Conversely, we saw very carefully structured SOST at AA Electro in Southern 
California. This plant assembles electronics products. Assemblers worked on their 
own to assemble an entire unit. Assemblers had to be certified before they could 
work on their own. The training for certification was mostly SOST and was highly 
structured. All the skills and procedures needed to assemble a particular product 
were documented and arranged in a logical order. Trainees received instruction 
fi'om the SOST instructor and were paired with a buddy who was a certified 
assembler. The trainees began with simple tasks and when those tasks were 
mastered, and their performance evaluated, they moved to the next-more-complex 
task until they could assemble an entire unit. Trainees regularly spent time with the 
trainers as they moved through the tasks. Careful records were kept and, at any 



er|c 



^ Conipaiiies who cooperated with the study were promised anonymity. Actual company names are not 
used in this report and company locations have been changed. 
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time, a supervisor could see what tasks and skills had been mastered, at what level 
the trainee was being trained, and what additional training was needed. 

The figure below is an example of a completely structured task from AA Electro’s 
training plan. 



Figure 10 









Review the workbench’s testing 
cables, their test function and how to 
perform a quality workmanship visual 
pre-inspection on these connectors 


Identify and explain the 
workbench’s testing cables and 
their functions; 

a) TTY-A 

b) Color video 

c) Keyboard 

d) SCSI 

Show and explain, using each 
cable’s connector, how to perform 
a quality workmanship visual pre- 
inspection on each connector, 
every time before its use. 

Ask the trainees to demonstrate 
their understanding of the 
workbench’s testing cables and 
their test function. 

Ask the trainees to demonstrate 
their understanding of performing a 
quality workmanship visual pre- 
inspection on each connector, 
every time before its use. 



In our survey of closed projects, only 33% of the project managers said their SOST 
training included trainees meeting individually, or in small groups, with SOST 
instructors to complete structured exercises. Conversely, 67% said that trainees 
were simply observed by the SOST instructor and got help as needed. 
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Jacobs and Jones (1995, p. 88) argue that effectively structured on-the-job training 
has modules with the following features: 



Figure 11 







Title 


Clearly describes the topic of the module 


Rationale Statement 


Explains why this topic is included in training and what 
trainee can do with the skills 


Training Objectives 


Tells trainee what he or she can know or do when 
training is completed 


Trainee Prerequisites 


Knowledge, skills, and attitudes the trainee needs 
before entering training 


Training Resources 


Equipment, supplies, and materials needed for training 


Training Content 


Information needed about the topic. May be presented 
in wide variety of formats: articles, textbooks, photos, 
diagrams, Web pages, etc. 


Training 


Describes how training is to be delivered 


Performance Tests and 
Feedback Forms 


Performance tests, rating scales, or other structured 
methods for giving trainee feedback on their 
performance 



• Trained Trainers 



In our view, to be a qualified trainer, a person should have mastered the content of 
the subject to be taught and have had specialized instruction in how to train in a 
SOST setting. In our fieldwork, we found that the trainers in six often case-study 
sites met these criteria. The six included two training-agency sites with 
professional trainers and a third where the trainer was provided by a vendor. 
According to our trainee survey, trainees consistently rated the quality of the fully- 
trained trainers better than the quality of those who were not trained. For example, 
trainers fi-om training agencies were rated 4.9 and 4.4 on a five-point scale while 
trainers assessed as not fully trained were rated 3.4 and 3.1 on that same scale. 

The difference between fully-trained and partially-trained trainers was clear to us in 
our field study. At one manufacturing plant with fully trained trainers, trainees 
would be assigned a series of tasks to complete, based on a training plan. The 
instructor would ensure that the trainees knew how to begin the task and had the 
correct tools and parts. If the trainees got stuck, the trainer would ask them 
questions and help the trainees solve the problem themselves. In contrast, at a 
financial services firm, when a trainee had a problem with software, the trainer 
(who was a regular supervisor) would come over and tell the trainee how to solve 
the problem, sometimes going so far as to sit in the trainee’s seat and punch the 
keys himself The goal was clearly to solve the software problem and" only 
secondarily to build trainee skills. 
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Some companies seem to treat the issue of trained trainers more leniently in SOST 
than in the classroom. For example, one of the ten sites we visited had very strict 
requirements for classroom trainers. Classroom trainers had to meet three criteria: 

1. Be certified in the skills being taught, or be a “revered expert,” 

2. Be experienced, and 

3. Have received positive evaluations (if used as a trainer before) 

At the same site, however, SOST “mentors” were simply selected by trainees and 
were not required to meet any of these criteria. 

According to our survey of closed projects, slightly over half the projects reported 
that SOST was provided, at least in part, by regular supervisors; about 60% reported 
that they used at least some company trainers; and 30% used vendor-provided 
trainers. 



Jacobs and Jones (1995), in their book, suggest the selection and training of trainers 
for structured OJT is a key ingredient in training success. 

Structured OJT trainers have basic requirements in two areas: 

They must have adequate knowledge and skill in the task and they 
must have adequate knowledge, skills, and attitudes as a trainer. If 
trainers do not meet the requirements in both areas, the 
effectiveness and efficiency of the training are likely to be at risk. 

(p.74) 

The authors suggest that trainers typically need about 12 hours of formal training that 
should conclude with having each trainer deliver a lesson from the program in which they 
would serve as an instructor. The figure below provides Jacobs and Jones' suggested 
learning objectives for trainers. 



Figure 12 






state the features of SOST. 

Predict the organizational consequences of 
using structured and unstructured OJT. 

Demonstrate how to analyze tasks in 
which they have expertise. 

Demonstrate how to prepare various 
components of structured OJT modules. 

Demonstrate how to get ready to deliver 
structured OJT by making use of the 
appropriate instructional events. 

Apply criteria to evaluate the effectiveness 
of the their own training. (Jacobs, 1990) 



Instructor training should use a variety of instruction methods - video 
demonstrations, small group activities, and opportunities for supervised practice. 
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• Customized to Companies 

This is the criterion on which ETP projects came closest to meeting the model. 
Training at seven of the ten sites we visited was completely customized to the 
company. At these sites, trainees completed SOST using the same processes, tools, 
and materials they would use in their regular jobs. The three exceptions are 
interesting. The first was a training agency conducting a SOST-only Welfare-to- 
Work project where trainees came to the agency site to receive generic training on 
standard software. The second was another training agency where trainees worked 
in a lab setting. The third was an engineering-intensive operation involved in the 
design and manufacture of aerospace equipment. 

At the training agency conducting training on standard software, trainees were - as 
part of the SOST - given workbooks containing exercises that all trainees had to 
complete. These generic exercises comprised roughly 40% of total SOST hours for 
this particular course. Once the generic exercises were completed, most trainees 
would attempt to apply the lessons to projects relevant to their own companies. 

The engineering-intensive aerospace operation was attempting to deliver, via SOST, 
“soft” skills such as communication and negotiation skills. These soft skills are 
generic, meaning they are completely transferable fi-om one setting to another - it is 
not feasible to customize the teaching of these skills to a specific organization. 

These generic soft skills are in contrast to “hard” skills that can be easily 
customized to the organization. For example, SOST can be designed to increase 
workers’ ability to perform perfect welds on unique company products using unique 
company welding machinery. 

• Delivered On-Site 

All but two of the programs we visited were delivered at the work site with trainees 
on the clock. One exception was the above-mentioned training agency conducting 
the SOST-only Welfare-to-Work project. The other exception was also a training 
agency, cited above, at which we observed SOST sessions on a Saturday using 
computers in the training agency’s lab. A fundamental problem with this type of 
off-site SOST is that all computing environments involve unique software 
installations and unique settings. These sometimes make lessons learned on one 
system hard to apply to another system. For example, a toolbar that appears 
onscreen in the lab may not appear onscreen at the trainee’s workplace computer. 
Unless toolbar settings are a specific part of the training, trainees might require 
additional help to translate off-site SOST lessons to the workplace. 

• Valid, Reliable Skills Certiflcation 

Only three of the ten sites we visited had valid, reliable skills certification, where 
employee skill attainment was carefully measured against a reliable and valid 
standard. One example of valid reliable skill assessment was a finance-industry site 
at which trainees were learning new software for claims processing. Trainees 
worked under close supervision, each claim was checked for accuracy, and 
productivity was carefully monitored. Trainees were not certified until they could 
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process the desired number of claims in a certain time period with a specified error 
rate. 

Four sites had partial skills certification, which went beyond simply having the 
instructor sign off that skills were attained. For example, at one site, a well-trained 
vendor-provided trainer spent time at each trainee’s workstation on a regular 
schedule to make sure they had mastered the current unit of the curriculum - in this 
case, blueprint reading. 

At the remaining three sites, skill certification was very informal. Trainers, who 
were usually regular supervisors, would simply sign off on skill attainment based on 
their perceptions of the trainees’ performance on die job. 

Our survey of closed projects supports these observations. Many projects used 
multiple methods to determine if trainees had attained the required skills. The most 
common assessment techniques were informal. For example, 82% of managers said 
"trainers observe trainees work on the job to see if they are using the skills, 
informally." Almost three-quarters (72%) said that they assessed SOST by having 
"supervisors observe trainees work to see if they are using skills, informally." 

Forty- four percent of project managers reported that they did not formally assess 
the effectiveness of SOST. 

The proportion of managers reporting formal assessment of learning was much 
smaller. About one third reported that supervisors used a formal evaluation, and 
about one third reported using tests. 

Reliable, valid skills certification is not possible without a carefully structured 
training plan. If the knowledge, skills, or attitudes to be developed by training are 
not carefully defined, it is impossible to measure their attainment. Hence, lack of 
careful planning in many SOST projects precludes valid skill certification. If the 
objectives of training are clearly stated, then skill certification is possible. 
Essentially, skill certification should occur at two points - at the conclusion of each 
training module, to ensure that trainees have achieved the desired objectives; and 
upon completion of training, to certify that the trainee has achieved the overall 
objectives of the training. Figure 13 on the next page provides some examples of 
skill certification methods. 
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Figure 13 









standardized Tests 


Standardized tests are often available from equipment 
manufacturers, software suppliers or professional groups 
(such as the American Society for Quality). If these tests 
are tied to the training plan, they are effective skill 
certification methods 


Locally Produced Tests 


Local training managers can develop tests to measure 
trainees’ knowledge of concepts taught. 

For example, a test of product knowledge could be 
developed for customer assistance staff. After training in 
the feature and maintenance of the new product they could 
complete the test to see if they had mastered the required 
knowledge. 


Observation, Rating and 
Feedback 


Structured rating forms can be developed based on the 
learning objectives of the training. Instructors can rate 
each trainee’s performance and provide him or her with 
feedback. 

For example, if training is in meeting skills, the instructor 
could observe the trainee chairing a meeting, rate various 
aspects of his behavior (starting on time, following the 
agenda, summarizing etc), and provide feedback. 


Evaluating Work Products 


Instructors can review and evaluate work products to 
evaluate trainee learning. 

For example, in the case of meeting skills, the trainer could 
review agenda and meeting minutes. In the case of 
processing insurance claims instructors can review claims 
for mistakes, look for patterns in the mistakes, and provide 
feedback to the trainee on their learning. 
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Issue #3: Contract administrators focus on billing trainer hours rather than achieving 
competence 

SOST monitors typically spend a sigmficant amount of time reviewing paperwork consisting 
largely of logs of SOST trainer hours. Contract administrators and consultants have learned to 
expect this inspection of paperwork and, therefore, spend a great deal of time ensuring that the 
paperwork is complete and in order. The incentive system is clear to contract administrators: 
There are serious consequences for botching the paperwork but apparently no serious 
consequences for not conducting formal and rigid certification of trainee skill attainment. 

Consultants saw the paperwork as more burdensome than company training managers. Our 
Delphi s^ey included the statement, “Overall, SOST documentation is burdensome to SOST 
trainers. The median response was 4.0 on a l-to-5 scale (with 1 indicating “Not Burdensome” 
and 5 indicating “Extremely Burdensome”). On the other hand, as Table 5 indicates, most 
company managers found the paperwork procedures to be difficult at the beginning but they 
mastered them over time. Only about one out of five managers reported continued problems 
with paperwork throughout the project. 



Table 5 







Procedures were clear and easy to complete. 


16.3% 


Procedures were initially difficult but were mastered them over time. 


62.1% 


Procedures were confusing and difficult and were a continuing problem 
throughout the project. 


22.4% 



A contract administrator at one site visit estimated that he spent roughly two hours per day 
attempting to ensure that trainer hours were accurately logged. The consultant, who worked on 
this particular ETP contract for about 32 hours per week, claimed to also spend two hours per 
day trying to ensure that paperwork was accurate and complete enough to satisfy a monitor. In 
cases where paperwork seemed to consume a disproportionate amount of resources, we 
observed the following problems: 

• Trainers, company supervisors in this case, felt intuitively that on-demand 
troubleshooting should not qualify as SOST. The trainers assumed that, as they were 
merely doing their jobs when supervising employees on the line, ETP reimbursement 
shouldn’t apply. The site’s ETP contract, however, allowed on-demand troubleshooting 
to qualify ^ SOST. The feeling that “I’m not training; I’m just doing my job as 
supervisor seemed a common mindset of supervisors in a manufacturing setting. 

• Our observations confirmed that trainers (company supervisors) were extremely busy 
during the workday, with rarely a minute fi-ee; therefore, the supervisors would typically 
wait until the end of the workday, or even another day, to fill out the paperwork. One 
trainer, reflecting a t)pical practice, indicated that he keeps pre-printed forms in his 
pocket but finds it difficult to remember to fill them out. This particular trainer felt quite 
certain he records only a fi-action of the SOST he performs. A long history of research 
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shows that when there is a delay, the recording of action is almost certain to be riddled 
with inaccuracies and omissions. 

While significant resources are being consumed in an honest attempt to tackle the “harder-than- 
it-looks effort to log trainer hours, the effort to track trainee skill attainment is comparatively 
weak. Of the 50 closed-project contract administrators surveyed, 38% indicated that they did 
not formally assess the effectiveness of SOST. Even in instances where on-the-job observation 
was used to assess trainee skill attainment, less than one-third of the observations resulted in 
formal assessment of trainee skill level; the other two-thirds involved some sort of informal 
assessment. 

We investigated two other aspects of SOST paperwork: the use of consultants and the role of 
the ETP monitor. We found that two-thirds of the companies had used a consultant to help with 
the project’s administrative work. As Table 6 shows, more than two thirds of the managers rated 
their consultant's performance "Excellent" or "Good," with about one fifth giving a rating of 
“Fair” or “Poor,” indicating some dissatisfaction. We observed one benefit that consultants 
brought to projects: various customized software programs that could be used for tracking 
SOST hours and generating reports to be signed by instructors and submitted to ETP. None of 
the sites we visited were using ETP’s recently implemented on-line system for recording SOST 
hours, so we were unable to assess its value. 



Table 6 



IManagers^pati^^^ 




.SPercent-lofe 


Excellent 


57.6% 


Good 


12.1% 


Average 


12.1% 


Fair 


3.0% 


Poor 


15.2% 



We were also interested in knowing how much help managers thought they got fi-om their ETP 
monitor. We asked each contract project manager to characterize his relationship with the 
company’s monitor as "Helpful," "Neutral," or "Unhelpful." Table 7 on the next page shows 
that ratings were generally very positive, with over 80% characterizing their monitor as helpful. 
This matched our field observations - we foimd that most monitors were supportive and 
working hard to make projects successful. Few monitors seemed to take an adversarial role 
towards contractors. 
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Table 7 









Helpful: 

Project monitor provided valuable advice and behaved in a 
professional and helpful manner. 


81 .6% 


Neutral: 

Project monitor was neither particularly helpful, nor difficult 
to work with. 


14.3% 


Unhelpful: 

Project monitor was difficult to work with and did not 
provide helpful service. 


4.1% 



Issue #4: Reliable Monitoring of SOST Is Difficult 

ETP uses monitoring to find out whether training at funded sites complies with legal, 
regulatory, and contractual expectations and requirements. Monitors are the eyes and ears of 
the ETP. However, the monitoring of SOST presents difficulties that make monitors unable to 
evaluate the actual delivery of SOST in the field. As a result, the monitoring of SOST yields 
little more than an assessment of the thoroughness of project documentation. 

Ideally, monitors would be able to validate that each of the elements of quality SOST is present. 
Monitors would routinely review SOST plans and schedules, and observe and assess SOST 
performed in the workplace. They would verify that SOST trainers have the requisite skills and 
that SOST is clearly linked to both the business of the firm and to corresponding classroom 
training. Finally, monitors would be able to determine whether trainees achieved the expected 
skill levels. 

Currently, monitors rely on interviews with key project personnel, trainers, and trainees, and on 
reviews of project documentation. While interviews can be very useful, the core of effective 
monitoring is documentation review and validation, direct observation, and analysis. These 
functions are hampered by a number of factors, as follows. 

• Lack of Structure in Training 

All the monitors we talked to admitted that they found it very difficult, if not impossible, 
to observe the delivery of SOST. None of the monitors at the ten case-study sites had 
observed SOST being given during their monitoring visits. In our visits, we also found it 
difficult to observe SOST even though we coordinated with the project managers in 
advance. The main reason for this is the ad hoc manner in which SOST is typically given. 
As discussed elsewhere in this report, structure is lacking in most of the SOST projects we 
reviewed. If there is no SOST schedule, monitors are unable to schedule their visits to 
observe SOST. If SOST has no definite curriculum, monitors often cannot differentiate 
beUveen SOST and routine supervisor oversight. At one site, for example, we interviewed 
a SOST trainer and expected to see SOST being delivered. The trainer explained that most 
of the SOST he gave was solving problems raised by trainees and none were raised while 
we were there. Consequently, we were unable to observe SOST - monitors often confront 
the same obstacles to SOST observation. 
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The lack of structure also adversely affects the usefulness of the interviews monitors 
conduct. Several monitors reported that trainers and trainees often do not understand what 
SOST is and confuse it with their routine duties. We had similar experiences in interviews 
conducted at several sites. When monitors ask trainees and trainers questions about 

SOST, they get responses more appropriate to the employer’s on-the-job training program 
than to SOST. 6^6* 

When SOST is not structured, monitors have difficulty in determining how, or if, it is 
related to ETP-funded classroom training. If SOST were given within a defined 
curriculum, monitors could reasonably assess how much it reinforces classroom training. 

• Decentralized Delivery 

SOST projects for Welfare-to-Work clients pose particular problems for monitors. These 
projects give trainees one-on-one training at employer and trainer sites in different parts of 
California. As a result, monitors face logistical problems in observing SOST. In addition, 
the SOST curricula are often unique to each trainee; to evaluate the project as a whole, 
monitors must evaluate each trainee’s experience. 

• Incomplete or Perfunctory Documentation 

Monitors rely heavily on reviewing SOST documentation, but the documentation is often 
incomplete, untimely, or perfunctory. Monitors focus their reviews of SOST on the 
documentation of trainer hours. This documentation is the basis for SOST reimbursement 
by ETP. Monitors for half of the projects in our sample indicated that this documentation 
was not up to date; this was corroborated by interviews with project managers and 
trainers. The most common reasons given were that employers did not understand SOST 
documentation requirements and that trainers found it difficult to remember to fill out the 
forms. As a result, forms are often completed weeks or months after the SOST was given. 
These delays make the monitors’ validation of documentation problematic. 

Several monitors we interviewed indicated that the documentation of trainee skill 
certification was perfunctory. Since these certifications are usually based on the 
attestations of supervisors, monitors generally have no way to validate them. Monitors 
must be satisfied with merely ensuring that the documentation has been completed. If 
certification were more objective (e.g. third party certification), monitors would have more 
reliable evidence of skill attainment. 

• Over-Reliance on Documentation 

SOST documentation alone is neither satisfactory nor sufficient. Its mere existence does 
not ensure that SOST has happened or that it was effective. Monitoring should go beyond 
the review of documents to determine the nature of the SOST actually being delivered and 
to assess its quality. This is only possible if SOST is clearly defined and differentiated 
within the employer s context, and if the skills of both trainers and trainees can be 
objectively measured. 

9 Monitors are Dissatisfied With Their Ability to Monitor SOST 

^ noted in the methods section, we conducted a focus group with project monitors, 
irfterviewed monitors in the field at each of the ten field sites, and interviewed managers 
oi ETP monitors. We found monitors themselves were dissatisfied with their ability to 
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effectively monitor the quality and quantity of SOST. They are keenly aware that they are 
often reduced to reviewing paperwork and interviewing a few trainees. They would much 
prefer to evaluate SOST "in action.” Because of this dissatisfaction, a number of monitors 
advocated discontinuing SOST, and they sometimes try to discourage contractors from 
including it in projects. 

Issue #5: SOST Reimbursement Is Often Unrelated To Actual Costs 

Our general finding is that the $80 hourly trainer reimbursement rate is probably too high in 
some cases and too low in a few cases; overall, there is a wide variation in the actual costs 
incurred. We found little evidence to warrant raising the rate. 

We used two different survey approaches to estimate the cost of SOST training. To estimate 
companies’ cost of in-house SOST training, we included cost-related questions in the telephone 
survey. To estimate consultants’ cost of providing SOST training, we included cost-related 
questions in fte modified Delphi survey. These two sxuweys provide interesting results, 

summarized in Figure 14 on the next page, regarding the actual costs of providing SOST 
training. 

The consultants’ estimates of the hourly rate for trainers have a median value of $50.00, a mean 
value of $52.n, and a 95% estimate band of $38.72 to $65.61. The consultants’ estimates of the 
“fully loaded” hourly rate have a median value of $101, a mean value of $102.67, and a 95% 
estimate band of $87.47 to $1 17.87. The fully loaded rate was calculated as the full cost of 
operation, plus profit, divided by the number of instructor hours delivered. 

The companies’ estimates of the hourly rate for trainers are much lower than the consultants’ - 
they have a median value of $29.50, a mean value of $40.75, and a 95% estimate band of 
$28.32 to $53.18. 
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Figure 14 

Instructor Hourly Rate 
(95% Confidence Interval and Median) 





The companies’ fully loaded rate estimates included one respondent who indicated that the fully 
loaded hourly rate for SOST training was $1000. Including that single outlier, the companies’ 
cost estimates had a median of $42, a mean of $83.96, and a 95% estimate band from $16 to 
$152. Without the outlier, the companies’ fully loaded cost estimates had a median of $41.42, a 
mean of $51.25, and a 95% estimate band from $37.99 to $64.15. 



With the outlier, the companies’ fully loaded estimates are close to the consultants’; in fact, the 
companies’ 95% estimate band overlaps the consultants’ 95% estimate band. Excluding the 
outlier, which seems appropriate, we found that the companies’ fully loaded rate was 
statistically the same as the consultants’ basic rate. 



While the consultants’ fully loaded cost was in excess of the $80 state reimbursement, the 
compames estimates of both basic and fully loaded rates were much less. Indeed, the $80 rate 
did not fall within the 95% estimate band. Although the companies’ average estimate was about 
$84 with the outlier included, this would be a misleading justification for increasing the $80 
rate, as the $80 would not begin to cover the $1000 high estimate but would more than 
adequately cover the total cost estimated by the majority of companies. 

We asked companies the cost question in one other way: “What percent of the total costs of 
SOST did the $80 per trainer hour cover?” The median answer was 75%. Interpreting this in 
relation to the fully loaded hourly instmctor rate estimates is difficult, but we can only assume 

that, when considering "total costs," managers assumed some costs not included in the "fully 
loaded" rate. 



On the Delphi survey we fried to gain some insight into what would happen if the Panel 
changed the price of SOST. We asked, "What is the lowest hourly ETP reimbursement for 
which SOST would still be included in contracts?” The 95% estimate band was $36.52 to 
$73.48, with a median of $45 and a mean of $55. 
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Figure 15 



Lowest Hourly ETP Reimbursement for Which SOST Would Still Be 

Included in Contracts 

(Range and Median) 




Lowest Hourly Rate 



High 

• Median 
Low 



In a similar vein, we asked the managers in the survey of closed projects what they would do 
differently if the reimbursement rate was lowered to $40 per instructor hour or raised to $ 1 60. 
Interestingly, Table 8 shows that, under both scenarios, a slight majority of managers said they 
would do the same amount of training. This inelastic response is probably due to managers’ 
views that the reinforcement of training on the job is essential and must be done whether or not 
incentives are offered. 



Table 8 



^^Mapagers,/R«§ponse,t(l'Cha)iiqei)ii 




ifs,emept:#atef^; 


^ jehanifv'^”' t j J8 






Amount of SOST would decrease 


47% 


7% 


Amount of SOST would stay the same 


53% 


56% 


Amount of SOST would increase 


0% 


37% 



We performed a final cost analysis by comparing the reported costs of projects we classified as 
structured, quasi-structured, and unstructured. Figure 16 on the following page indicates that the 
median cost of a fully loaded instructor hour was almost identical across the three project 
groups, varying only from $42 to $44. Interestingly, the 95% confidence bands around the 
median were much larger for the unstructured and quasi-structured projects. This indicates to us 
that the current rate is adequate to support fully structured SOST. 
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Figure 16 
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Based on these survey results, the current $80 hourly reimbursement for SOST exceeds the 95% 
estimate band of the minimum compensation level for which SOST would still be included in 
projects (Figure 15). The results also indicate that it would take a substantial increase or 
decrease in the reimbursement rate to change the amount of SOST employers are willing to 
provide. However, these results relate to the current models of SOST - it is reasonable to 
believe that the estimate band, as well as other cost estimates, would shift upward if additional 
resources were required to more fully customize and structure on-site training. 

Issue #6: SOST-only Projects Pay More Than Market Price For One-on-One Counseling 
and Generic Classroom Training 

We were able to visit one SOST-only project run by a training agency, serving former welfare 
recipients who were now employed. We also interviewed ETP staff about these projects. 

• How Training Works 

The project we visited essentially enrolls former welfare recipients into ETP and offers 
them generic training on various types of office automation software, including Word, 
Excel and Access. Trainees also get some life planning and other soft-skills training. 
Training is individual or in groups of two or three. The agency offers instruction in two 
languages other than English, as many trainees are recent immigrants. 

At the time of our visit, the trainees were already employed. They came to the training 
agency when they were off work or released from work. The training they received might 
or might not have been related to their ciirrent jobs. Frequently, trainees are seeking 
training to get a better job. For example, one woman we interviewed worked in a local 
retail store afternoons and early evenings. She came into the training agency to get 
training in Access and computerized bookkeeping in hopes of ultimately getting a job as a 
bookkeeper. She received one-on-one instruction for two hours and left. 
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• How Funding Works 

The SOST funding formula is as follows. If, for example, the average number of trainee 
SOST hours is 80 and if there are 100 trainees, the total SOST hours are calculated to be 
8,000. The formula assumes one SOST instructor for ten trainees, so the number of trainer 
hours to be reimbursed is calculated by dividing the total number of trainee hours by ten, 
in this case gelding 800 trainer hours to be tracked by ETP. The contract will therefore 
deliver 800 instructional hours. Since many trainer hours will be spent with one, two or 
three trainees, the hours an individual trainee receives will be far fewer than the planned 
80. If our hypothetical trainer always works with ten trainees, each trainee will get 80 
hours of training, but if the instructor always works with two trainees, each trainee will get 
only 16 hours of training, and if the trainer always works one-on-one, each trainee will get 
only eight hours of training. 

In the case of the above-described project, the proposal officially called for 700 hours of 
SOST instruction, but since this number is just used to calculate trainer hours, far fewer 
hours are received by any particular trainee. If all training were one-on-one (as it often 
was), the trainee would receive only 70 hours of instruction. The agency is reimbursed 
$80 for each hour the instructor teaches. Instructors are paid between $25-$35 at this site; 
most work part time and do not receive fiinge benefits. This arrangement is typical of 
other SOST-only Welfare-to-Work projects. 

We find ETP’s rate of reimbursement to be far above the market cost of similar training. 
One can receive, for example, 100 class hours of training in Microsoft office applications 
for far less than $8,000. Another comparison is that an hour of therapy with a licensed 
psychologist commonly costs $100 or less. Training agencies have found that it is 
profitable to provide one-on-one standard training under SOST, as they are reimbursed 
$80 per instructor hour. Training of small groups is not as profitable at the regular ETP 
cl^sroom reimbursement of $13 an hour. For example, assume an agency had four ETP- 
eligible MS Word trainees. If this were treated as classroom training, the agency would 
receive only $52 per instructional hour rather than $80 if the training were classified as 
SOST. Agencies argue that one-on-one training is actually efficient because students learn 
more with one-on-one instruction. This may be true, but instruction would have to be 
more than six times as efficient to warrant the higher cost. In other words, for one-on-one 
traimng to be as efficient as regular ETP classroom training, a trainee would have to learn 
as much in 20 hours of one-on-one instruction as they would learn in 120 hours of regular 
classroom instruction. 
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Policy Options and Alternatives 

In this section, we present a range of policy alternatives for the Panel’s consideration, with pros 
and cons based on our research. We then present the financial implications of each alternative. 
In the final section, we present our recommendations. 

SOST Policy Alternatives 

• Alternative!: No Change in SOST 

TWs policy alternative would continue funding SOST projects at $80 per trainer hour, 
with a ratio of two hours of SOST to one hour of classroom instruction; this has been the 
policy since 1996. The advantages of maintaining status quo are that this alternative costs 
least in time and effort, and avoids the controversy that might arise in implementation of 
other policy options. Furthermore, our research indicates that even companies using 
informal, unstructured SOST report benefits. 

However, maintaining status quo does not solve the problems, outlined in our analysis, 
which may worsen with time. For example, contractors will continue to “game the 
system by funding existing training with ETP dollars; less effective unstructured on-the- 
job training will likely continue to be the dominant mode of SOST, reducing the potential 
return on ETP's investment in SOST training; and monitors will continue to be frustrated 
at their inability to adequately monitor SOST. 

• Alternative 2: Require that contractors have a plan to reinforce classroom training, 
only fund SOST training that meets new and rigorous structural criteria, and stop 
funding SOST-only projects. 

This alternative takes into account what research has shown - on-the-job reinforcement is 
essential for classroom training to have a real impact on productivity. The policy would 
have four elements: 

1. Contractors would be eligible for SOST reimbursement if they have plans that 
include the five elements deemed essential to structured-on-site- training'*. Again, 
these elements are: 

A Structured Plan: 

Plan includes sequenced modules, measurable training objectives, and specific 
on-the-job activities tied to class instruction. 

Trained Trainers: 

Trainers are trained in both content and instructional methods. 

^ Customization to Companies: 

Traimng uses company's products, tools, machines, and procedures. 

Delivery On Site: 

Training is done at the workplace and on the clock. 

Valid, Reliable Skills Certification: 

There is objective, frequent measurement of skill attainment tied to job 
performance. 



A more detailed description of how these elements would be used in ETP SOST training is included in the results 
section under Issue #2. 
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2. Contractors would receive the current $80 hourly instructor reimbursement, up to the 
current ratio of two hours of SOST per classroom hour. The policy of tracking 
instructor hours, not trainee hours, would be retained. 

3. Structured training would be much easier to monitor because it is scheduled and 
therefore can be observed. Valid and reliable skills certification would give monitors 
better evidence than a supervisor’s sign-off to verify skill attainment. Tightening 
quality standards would discourage “gaming” of the system and reduce funding for 
training that would have taken place even without ETP funding. The overall quality 
of the training should improve, enabling contractors to realize the full potential 
benefit of training. 

4 . SOST-only projects would not be funded. They are an expensive, inefficient way to 
deliver generic training that could be conducted in a classroom or lab setting. 

On the downside, additional conditions for funding would increase the complexity and 
cost of the application process. Contractors would need to develop their training plans and 
certification programs before applying for funding, or they might be required to have a 
certification process in place prior to funding. This definitely could be to the disadvantage 
of small companies that do not have the upfront resources to comply with these funding 
demands. This alternative would also increase the monitoring burden for ETP staff and the 
contractors themselves. 

• Alternatives: Eliminate SOST 

In this case, ETP would simply cease funding SOST. It would only fund class and lab 
instruction. 

This alternative has the advantage of removing the risk of paying for low-quality training 
or training that would happen even without ETP funds. It also frees up resources for more 
classroom or lab training and “releases” monitors from trying to track the “elusive” SOST. 

On the negative side, discontinuing SOST eliminates a powerful and proven training 
intervention provided by ETP. Over two thirds of trainees surveyed at our case-study 
companies rated SOST “extremely valuable” in improving their skills. Project managers 
report that SOST, de^ite its flaws, added as much value as class or lab trai nin g 
Furthermore, eliminating SOST would engender much controversy, because it has been a 
part of the ETP program from its earliest days. 

Financial Implications of Policy Options 

Different SOST policies can be expected to yield different SOST expenditure levels. The 
financial implications of the three SOST policy alternatives are discussed below. 

• Alternative 1: No Change in SOST. 

Key Implication: SOST cost remains the same. 

ETP is spending 16.8% of its training dollars on SOST under current SOST policy. While 
the SOST percentage varies from year to year, the absence of any detectable trend implies 
that ETP can expect to to spend about this proportion of its training funds on SOST if 
policy is not changed. 
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However, if SOST-only contracts - those involving only SOST training and no classroom 
or lab hours - were eliminated as recommended above, about 28.5% of SOST 
expenditures would be eliminated from the training budget. To illustrate the budgetary 
implications of this policy move, suppose that ETP spends $60 million in a fiscal year on 
training contracts. Under present SOST policy, 16.8% of that, $10 million, would go to 
fimd SOST (one out of every six dollars). If SOST-only contracts were eliminated, SOST 
spending would fall by 28.5%, about $2.8 million, leaving SOST fimding at $7.2 million. 



Figure 17 






Hypothetical annual $60 million ETP training expenditure currently 
implies $10 million SOST expenditure. 

Alternative 1— no change in SOST funding— means $10 million SOST 
spending continues, imless SOST Only projects are dropped. In which case: 

Current $10.0 million SOST expenditure 
minus $ 2.8 million SOST Only project evpendinire 
$7.2 million SOST expenditures remaining 

Net savings: $2.8 million if SOST Only projects dropped. 



• Alternative 2: Require that contractors have a plan to reinforce classroom training, 
only fund SOST training that meets new and rigorous structural criteria, and stop 
funding SOST-only projects. 

Key Implication: SOST cost is reduced by half. 

Restricting funding to fully structured SOST will reduce SOST spending by about 70%, 
but some increased class and lab spending will mean that net savings to ETP will be only 
around 50% of its current SOST expenditures. 

To estimate the impact of this policy, we used information from our survey of 50 closed 
projects. We evaluated the “degree of structure” in the SOST component of training 
according to the five elements specified under Alternative 2 above. We found that the 
degree of stmcture varied considerably - some SOST components had all five elements 
and were considered completely structured, others had some elements of structure, and 
some had none of the five elements and were considered unstructured. Five of the 50 
projects were completely structured, 23 had some elements of structure, and 22 were 
completely unstructured. In the middle group of 23 SOST contracts, 9 had 4 of 5 desired 
structural characteristics, 10 had 2 or 3, and 4 had only 1 of the desired characteristics. 

Presuming that these 50 contracts are representative of the usual array of SOST projects, 
distribution by their degree of structure can be used as a guide to estimate the likely 
funding consequences of more restrictive SOST standards. We grouped these 50 contracts 
according to their specific SOST characteristics, and each group ’s share of SOST funding 
was determined. The results are presented in Table 9 on page 46. (Note that we dropped 
one of the projects in the “Mostly Stmctured” group because it was a SOST-only project 

The Role of Structured-On-Site-Training In ETP - Final Report 44 



50 



H5I17QT mpv AVAILABLE 



Management and Organization Deveiopment Center. Caiifornia State University, Northridge 



and therefore would not be funded under Alternative 2. Besides, because that project was 
very richly funded, it would greatly distort the distribution of funds over the groups). 
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Our research suggests that SOST funding would drop by more than half in the face 
of a requirement that SOST be completely structured to qualify for ETP support. 
Still, we believe a sizable segment of SOST contractors would elect to meet the 
new standard. Obviously, those who already have completely structured SOST 
would qualify for funding. We also believe that most, if not all, of the middle 
group - those with some or most of the elements of structure - would elect to 
provide completely structured SOST for two reasons. First, the new standards 
would provide a clear blueprint for what ETP would expect of SOST in order to 
qualify, and some in this group would have to add very little additional structure in 
order to qualify. Second, our research shows most SOST providers estimate the 
cost to be less than $50 per trainer hour, which implies that they could add 
structural elements at some additional cost, and SOST would still be worth doing if 
compensated at $80 per trainer hour. However, we believe that those with 
imstructured SOST, representing over half of regular SOST spending, would not 
add the necessary structure to qualify for SOST support, because of the high cost of 
structuring what is now mostly trouble-shooting by supervisors. We applied this 
logic to the survey results in Table 9. 

The first group consists of the five “Completely Structured” projects that represent 
10.2% of the contracts and 10.3% of the SOST funding in the sampled contracts. 
These projects would be expected to receive SOST funding under the new, stricter 
standards. The eight “Mostly Structured” SOST projects had most of the elements 
required of completely structured projects. This group would require little 
modification to be funded under the stricter standards, and the funding incentives 
would likely induce the necessary modifications. This group represented 1 6.3% of 
the sample contracts and 15.7% of sample SOST funding. Together the completely 
structured and mostly structured contracts represented 26.0% of sample contracts 
and 26.5% of funding. If the completely structured projects were funded under the 
stricter standards, and the mostly structured projects were induced to add the 
necessary elements to be funded, and all other projects were excluded, about 25- 
30% of current SOST contracts would continue. The percentage of continued SOST 
funding also would be 25-30%. 

The “Less Structured Group” had only one or two of the elements required of 
structured SOST. These contracts would require more modification to qualify for 
funding imder stricter standards, but we believe that the financial incentive of $80 
per trainer hour for SOST funding would induce most in this group to qualify. The 
14 contracts in this group represent 28.6% of the sample contracts but only 15.3% 
of total sample SOST funds. If this group responded to the funding incentives, 
along with the more structured groups, it would push the percentage of contracts to 
55% and the percentage of SOST funding to just over the 41%. 

It is much less likely that many in the unstructured group would respond to the 
funding incentives, because the SOST support level would stay where it is now, but 
the required change in contract structure would be significant. These contractors, 
who now use supervisors in a trouble-shooting role to deliver SOST, would have to 
provide all the required elements of SOST structure, including trained trainers. We 
do not believe many companies would consider this dramatic change to be 
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worthwhile. Our analysis suggests that imposing a requirement of complete 
structure to qualify for SOST funding would result in up to 55% of the current 
number of SOST contracts and just over 40% of the current SOST funding level, in 
the future. If the policy change were accompanied by any increased support for 
SOST, more contractors could be expected to respond, increasing both the number 
of contracts and the SOST funding levels above these projections. Note also that 
these projections are based on previous contracts absent the stricter standards. In 
the future, with a set of well-defined standards and a growing number of people 
experienced in meeting those standards, the number of contracts and approved 
funding levels could grow beyond these predicted levels. 

This discussion above implies that imposition of the completely structured SOST 
option would free almost 60% of the currently allocated SOST funds to support 
other ETP training activities. Our further analysis suggests that the savings would 
be considerably less. The average trainee, in a training job with SOST, costs ETP 
$631 in SOST support dollars, based on analysis of completed contracts. However, 
the difference between the average per-trainee cost in a job with SOST and one 
without SOST is only $349. This smaller-than-expected difference results from a 
higher average number of class or lab hours in jobs without SOST compared to 
those with SOST. We believe that contracts without funded SOST would substitute 
some additional class or lab hours that would otherwise have taken place as SOST 
learning. 

This analysis suggests that the actual savings to ETP from the stricter SOST 
standards option would only be about 55% of the estimated decrease in SOST 
support. That is, the expected reduction of nearly 60% in SOST funding would 
imply only a 33% savings. 

To illustrate the financial implications, consider our previous illustration with ETP 
spending $60 million a year on training and SOST support at $10 million. 
Elimination of the SOST-only contracts would reduce SOST support by $2.8 
million, leaving $7.2 million in SOST spending. We estimate that imposing the 
stricter SOST standards would reduce the remaining $7.2 million SOST support by 
just imder 60%, which would reduce SOST spending by $4.2 million, from $7.2 
million to $3 million. That is, SOST spending would be reduced from $10 million 
to $3 million under Alternative 2, but increased class and lab costs would absorb 
some of the $7 million in SOST savings. 

Increased class and lab hours in the contracts that drop SOST funding would reduce 
net savings to only 55% of the $4.2 million in SOST funds that they represent. The 
other 45% would go toward class and lab spending in these contracts, pushing those 
costs up by $1.9 million. 

In siunmary, using our $10 million SOST example. Alternative 2 would decrease 
SOST spending by $7 million to $3 million, but class and lab spending would rise 
by $1.9 million, yielding net savings of $5.1 million. The good news for ETP 
monitors is that the remaining $3 million in SOST components would be fully 
structured and have scheduled time, structured exercises, trained trainers, and 
formal skill attainment evaluations, ail of which make SOST much easier to 
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monitor. The additional $1.9 million in class and lab costs would be part of 
contracts that are already monitored and may not require extra monitoring time. 



Figure 18 






Hypothetical annual $60 million ETP training expenditure 
currently implies $10 million SOST expenditme. 






Alternative 2: Fund only Completely Structured SOST projects, 
Drop SOST Only projects. 

Current $10.0 million SOST expenditure 
Minus $2.8 million in SOST Only projects 
Minus $4.2 million Unstructured SOST projects 
$3.0 million in Structured SOST projects 

Plus $1.9 million in additional class/lab costs in 
currently Unstructured SOST projects 



Net Savings $5.1 million 



• Alternative 3: Eliminate SOST 

Key Implication: SOST cost is reduced by more than two thirds. 

ETP would shift some costs from SOST to class and lab training, resulting in net 
savings of about 68% of current SOST costs. 

The third alternative - eliminating SOST funding - would seem to have clear 
financial ramifications: SOST funding would disappear from ETP budgets, and 
approximately 16.8% of current training dollars would become available to support 
other training. As noted, however, training contracts without SOST tend to have 
more class or lab hours than contracts with SOST, and so net savings in training 
support would be less than the reduction in SOST funding. While the average SOST 
cost per trainee is $631, the average difference in the per-trainee costs of projects 
with SOST and without SOST is only $349, or 55% of the SOST costs. This means 
that eliminating SOST would save 55% of the previous level of SOST support. 

The financial implications of eliminating SOST funding can be illustrated by using 
again our example of $60 million of ETP spending on training with $10 million 
allocated to SOST. Eliminating SOST-only contracts reduces SOST spending to 
$7.2 million, as previously explained, and banning all SOST support would 
eliminate this balance. However, with SOST banned, we believe that the class and 
lab hours would grow in these contracts by 45%, or $3.2 million, of the SOST 
savings. The net savings to ETP of eliminating $10 million in SOST support would 
thus be $6.8 million. 
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While this alternative would eliminate SOST monitoring, it would increase class 
and lab costs in some existing contracts by $3.2 million, thereby adding some 
monitoring expense. However, as the class and lab components of those contracts 
would be monitored anyway, the increased costs should be minimal. 



Figure 19 




Hypothetical annual $60 million ETP training expenditure 
currently implies $10 million SOST expenditure. 

Alternative 3: Ban SOST funding. 

Current $10.0 million SOST expenditure 
Minus $2.8 million SOST Only projects 
Minus $7.2 million in other SOST projects 
$0 in SOST expenditures 

Plus $3.2 million in increased class/lab expenditures in 
the unfunded other SOST projects 

Net Savings $6.8 million 
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Recommendations 

1. Require that contractors have a plan to reinforce classroom training, but only 
fund SOST training that meets new and rigorous criteria for being structured. 

We recommend that the Panel adopt this alternative. It recognizes the value of 
reinforcing classroom training in all projects, but restricts funding to the highly 
structured on-site training that research suggests offers the most added value for ETP’s 
investment. Projects that have a high degree of structure are much easier to monitor 
and therefore solve many of the problems in the existing arrangement. 

Perhaps most importantly, this policy ends the practice of supervisors providing 
informal coaching as part of their regular duties and charging it to EXP as SOST. In 
those situations, EXP funds clearly do not supplement existing training, the quality of 
training is relatively poor, effective monitoring is virtually impossible, and positive 
outcomes are unlikely. 

2. Eliminate SOST-Oniy Projects 

We recommend that the Panel move from its current moratorium on funding new 
SOST-only projects to permanently eliminating such funding. These projects are not 
structured on-site training but rather vehicles for providing generic training and 
counseling to individuals or small groups who caimot be served profitably under the 
classroom-training formula. As we note in our findings, we believe the cost of 
training under these projects substantially exceeds the market price for the training or 
counseling delivered. These projects represent a substantial portion of current SOST 
investment, which we estimate to be almost 30% of all SOST expenditures. 

If the Panel wants to serve groups such as Welfare-to-Work participants or employees 
of very small businesses, we recommend reconsidering the class-and-lab funding 
formula to provide incentives to serve these groups rather than continuing to 
effectively misclassify generic training and counseling as SOST. 

3. Keep the Existing $80 per Trainer Hour Rate and Track Trainer Hours 

We have argued that EXP has been over-paying for the informal coaching by regular 
supervisors, which makes up most of SOST, but we do not believe the current rate 
overpays for quality SOST that has the five key elements described previously. If ETP 
funds only this quality SOST, contractors' costs are expected to be higher. Instructors 
will have to be trained or hired. Compames will have to develop structured exercises 
to be done on the job. Production will slow during training, and assessing skill 
attainment will take time and resources. Since we did find a number of contractors 
who delivered fully structured or nearly fully structured SOST, we believe the current 
price is adequate support for a fully structured program^. 

4. Plan To Evaluate The Impact Of Any SOST Policy Changes 

All policies generate unintended consequences and may or may not achieve intended 
benefits. We recommend that whatever SOST pohcy the Panel adopts, they plan and 
implement systematic evaluation of the policy’s impact on skill attainment, ETP 
finances, and program administration. 



^ It is interesting, however, that structured projects do not report higher costs than unstructured projects. 
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Appendix A 

Consultants Interview Guide 



Date: 

Proj ect 

Interviewer: 

Interviewee: 

Company: 

Describe purpose of the project. (We are evaluating the SOST component of the ETP 
project. We are not here to evaluate your particular project. We just want to see how 
SOST is being implemented) Assure the interviewee about confidentiality. 

1 . How did you get involved with this company and this project? 

• What is your role in the project? 

• Have you had other ETP experience? 

• What were the primary goals for the project? 

• How has the project been working out? 

2. Were you involved in the decision to include SOST In this project? If YES, why did 
you decide to include a SOST component in this project? 

• What were the goals of SOST? How were they different fi-om those of the 
classroom component? 

• What did you see as the primary benefits of using SOST? 

• Does your company have other OJT activities? 

3. What would you have done if SOST support fi’om ETP had not been available? 

4. How did you originally plan to deliver SOST, and how did you actually end up doing 
it? 

• Is your SOST training provided by internal staff? If so, did they need to be 
trained? 

• Are you using a training agency or consultant(s)? If so, how was the consultant(s) 
or training agency selected? 

• How was SOST designed into the training schedule? 

• How did you decide how many hours of SOST to include? 

® Did you need any special training materials or equipment to support SOST? If 
yes, what did they cost? 
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I would like to ask a few questions about how you are actually implementing SOST. 

5. How do you schedule and deliver SOST? 

• If SOST is provided as needed, does the employee request training from the 
trainer, or does the trainer initiate the training? 

• Do trainers continue their regular duties as well as doing SOST, or are they doing 
SOST only? 

• How much are the trainers paid per hour? 

0 What successes and problems have you encoimtered in delivering SOST? 

6. ETP requires that you track instructor time and verify that employers have mastered 
the intended skills, how have you met these requirements? 

• What record-keeping procedures do you employ? 

• Do you feel these procedures accurately capture SOST results? 

• How would you characterize your relationship with ETP monitors? 

• What problems have you encountered? 

• How effective and efficient is the monitoring process? 

7. How effective has SOST been at accomplishing your workforce training goals? 

• Have it helped trainees gain skills? How do you know? 

• What has influenced the effectiveness of SOST? 

• How are the results evaluated? 

• Would you use SOST again for training? 

8. Do you feel the current SOST reimbursement rate of $8.00 per trainee is adequate? 

• How would you estimate the cost of SOST training at this company? 

• What factors would you consider? 

• How would you estimate the costs of these factors? 
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Appendix B 

Delphi Questionnaire 

SURVEY #1: Background Information & Opinions 
Please type your responses on a "reply" to this e-mail. 

Example; X is a marked answer. 

How many years have you (& your firm) provided ETP consultant services? 

How many ETP projects has your company completed to date? 

Of these projects, how many included an SOST component? 

How much do your projects pay SOST instructors per hour, salary only/ 
(Don't know )? 

How much do you pay in additional fiinge benefits? 

(Don't know ) ~ 

What do you estimate the "fully loaded" cost of SOST instructor time to be? 
(Please consider salary, fiinges, overhead, any other relevant costs). 



What is your estimate of the materials costs per trainee of SOST training? 
(Don't know ) 

Did SOST cause your clients to have any lost production? 

Yes 

(If YES); 

Please estimate the cost of the lost production 
No 

How did you estimate that cost? 



Were trainees always on the clock for SOST? 

YES 

Don't Know 

NO (If NO); What percent of the time were employees on the clock? % 

What percent of your projects' total SOST costs would you estimate were covered by the 
SOST, $80 per instructional hour, reimbursement? % 
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What would your clients have done differently if the reimbiu-sement had been $40. 00/per 
horn- 

Training would have decreased 

Training would have stayed the same 

- Training would have increased 

What would yoiu- clients have done differently if the reimbiu-sement had been 
$ 1 60 . 00/per hour 

Training would have decreased 

Training would have stayed the same 

Training would have increased 



ADMINISTRATION 

Which statement best describes your clients' experience completing the procedures 
required to keep track of SOST hours? 

1 Procedures were clear and easy to complete 

2 Procedures were initially difficult but we mastered them over time. 

3 Procedures were confusing and difficult and were a continuing problem 

throughout the project. 

If #3, please elaborate: 



What do you see as the most serious administrative problems you have experienced with 
SOST? 
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Appendix C 

Internal Project Manager Interview Guide 



Date:^ 

Proj ect 

Interviewer:^ 

Interviewee: 

Title:^ 

Describe the purpose of the project. (We are evaluating the SOST component of the ETP 
project. We are not here to evaluate your particular project. We just want to see how 
SOST is being implemented). Assure the interviewee about confidentiality. 

1 . How did your company get involved with the ETP program? 

• How did you first learn about ETP? 

• Had the company had earlier ETP proj ects 

• What were the primary goals for the proj ect? 

• How has the project been working out? 

2. Why did you decide to include a SOST component in this project? 

• Did a consultant play a role? 

• What were the goals of SOST? How were they different from those of the 
classroom component? 

• Does your company have other OJT activities? Please describe. 

3. What would you have done if SOST support from ETP had not been available? 

4. How did you originally plan to deliver SOST, and how did you actually end up doing 

• Is your training provided by internal staff? If so, who are they and did they need 
to be trained? 

• Are you using a training agency or consultant(s)? If so. How was the 
consultant(s) or training agency selected? 

• How was SOST designed into the training schedule? 

• How did you decide how many hours of SOST to include? 

• Did you need any special training materials or equipment to support SOST? If 
yes, what did they cost? 
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I would like to ask a few questions about how you are actually implementing SOST. 

5. How do you actually schedule and deliver SOST? 

• If SOST is provided as needed, does the employee request training from the 
trainer, or does the trainer initiate the training? 

• Do trainers continue their regular duties while they are doing SOST, or are they 
doing SOST only? 

• How much are the trainers paid per hour? 

® What successes and problems have you encountered in delivering SOST? 

6. ETP requires that you track instructor time and verify that employers have mastered 
the intended skills. How have you met these requirements? 

• What record-keeping procedmes do you employ? 

• Do you feel these procedmes accmately capture SOST results? 

• How would you characterize yom relationship with ETP monitors? 

• What problems have you encountered? 

• How effective and efficient is the monitoring process? 

7. How effective has SOST been at accomplishing your workforce development goals? 

• Has it helped trainees gain skills? How do you know? 

• What has influenced SOST’s effectiveness? 

• How are the results evaluated? 

• Would you use SOST again for training? 

• How would you compare the effectiveness of SOST to classroom instruction? 

8. Do you feel the current SOST reimbursement rate of $8.00 per trainee is adequate? 

• How would you estimate the cost of SOST training to your company? 

• What factors would you consider? 

• How would you estimate the costs of these factors? 



Thank you for your help! 



D 
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Appendix D 

SOST Closed Project Interview Guide 



Date: 

Company Name: 

Interviewee: 

Title: 

Briefly describe the ptupose of the project: We are evaluating the SOST component of 
training. As part of our methodology, we are talking to managers of completed ETP 
projects to collect their opinions on how successful SOST was as a training tool. Your 
company will not be identified in our report and all information we collect will remain 
anonymous. Could we take 15 minutes right now to complete the survey? 

YES. Begin 

NO. What would be a good time to call back? (Schedule call-back) 



How many ETP contracts has your company completed to date? 

□ (1) One 

□ (2) Two 

□ (3) Three or more 

Of these contracts, how many included a SOST component? 

What were your goal(s) for SOST? 

□ (1) To help trainees apply what they learned in the classroom to day-to- 
day work 

□ (2) To provide structured practice of skills learned in the classroom 

□ (3) To provide on-demand help with problems when trainees returned to 
their job 

□ (4) To certify that employees have learned skills taught in the classroom 

□ (5) To provide one-on-one instruction. 

□ (6) Other (PLEASE LIST): 
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4. Please describe how you implemented SOST on your last project? (ASK OPEN- 
ENDED BUT CODE THE RESPONSE BELOW) 

Arrangement 

□ (1) Training agency provided class or lab (non-customized exercises) off-site 

□ (2) Training agency provided customized SOST (tailored to actual work 
projects) off-site 

□ (3) Vendor-provided customized SOST on-site 

□ (4) Company trainers provided customized SOST on-site 

□ (5) Company's regular supervisors provided SOST on-Site 

□ (6) Other (PLEASE SPECIFY): 



Pedagogy 

□ (1) Trainees had structured exercises to individually complete on-the-job. 

□ (2) Trainees met in small groups with instructor and completed structured 

exercises on the job. 

□ (3) Trainees were observed by SOST instructor and got coaching when 

needed. 

□ (4) Trainees asked for help when they needed it, and the instructor provided 

help. 

□ (5) Other: (PLEASE SPECIFY) 



SOST EFFECTIVENESS 

5. How effective was SOST in comparison with classroom tr aini ng in achieving 
your goals.... 

□ (1) not as effective as ETP classroom training 

□ (2) about equally effective as ETP classroom training 

□ (3) more effective than ETP classroom tr ainin g at 

□ (4) impossible to compare to ETP classroom training because the skills 
delivered are completely different 
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6. How much of the value added by training did you expect to gain from. . . 

(ASK RESPONDENT FOR A PERCENTAGE AND RECORD IN TABLE) 

7. How much of the value added did you actually gain from. . . 



Classroom 



SOST 



Expected Actual 



Contribution Contribution 



(6a) % 


(7a) % 


(6b) % 


(7b) % 



8. If ETP had not paid for SOST, what would your company have done? 

□ (1) Done the same amount of SOST training 

□ (2) Done some SOST training but less than done now 

□ (3) Done no SOST training. 

9. In this project which of the following do you use to assess the effectiveness of 



SOST? (ASK OPEN-ENDED AND THEN CODE) 





Used 


Did not use 


Supervisors observe trainees' work to see if they 
are using the skills, informally. 


1 


2 


Supervisors observe trainees work to see of they 
are using the skills, complete a formal 
evaluation. 


1 


2 


Trainers observe trainees work on the job to see 
if they are using the skills, informally 


1 


2 


Trainers observe trainees work on the job to see 
if they are using the skills, complete formal 
evaluation 


1 


2 


Have trainees complete specific projects or 
activities to see if they learned skills 


1 


2 


Questionnaires completed by trainees 


1 


2 


Tests administered to trainees 


1 


2 


We don’t formally assess SOST effectiveness; 
we assume it’s working 


1 


2 


Other 


1 


2 
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9. How would you rate the level of the following potential individual benefits from 
SOST? 





A major 
benefit 


A minor 
benefit 


Employees do not 
benefit in this way 


Increased opportunity for advancement in 
this company 


3 


2 


1 


Increased pay in this company 


3 


2 


1 


Learn skills that are valuable in the larger 
labor market 


3 


2 


1 


Increased interest and motivation in their 
job 


3 


2 


1 



10. How important was SOST in maintaining your company’s competitiveness? 

□ (1) Critical; we would not be able to compete without SOST 

□ (2) Very important; SOST increases our overall competitiveness 
substantially 

□ (3) Somewhat important; it seems to help marginally 

□ (4) Not important 

□ , (5) Don’t know (I can’t even begin to guess on this question). 

SOST COSTS 

1 1. How much do you pay your SOST instructor per hour, salary only? 



12. How much do you pay in additional fringe benefits? 

13. What you estimate the “fully loaded” cost of SOST instructor time to be? 



(Consider salary, fiinges, overhead, and any other relevant costs) 

14. How would you estimate the materials costs per trainee of SOST training? 



15. Did SOST cause any lost production? 

□ (l)Yes If Yes, (AS IQ: What was the cost of the lost production? 



□ (2) No 

15 a. How did you calculate that cost? 
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16. Were trainees always on the clock for SOST? 

□ (1) Yes 

□ (2) No If no, (ASK): What percent of the time were employees on the 

clock? % 

17. What percent of your total SOST costs would you estimate were covered by the 

SOST, $80 per instructional hour, reimbursement? % 



18. What would you have done differently if the reimbursement had been $40 per 

instructor hour? What about $160 per instructor hour? 



a. $40.00/ner hour 


b. $160.00/ner hour 


□ ( 1 ) Training would have 


a 


(1) Training would have 


decreased 




decreased 


□ (2) Training would have stayed 


□ 


(2) Training would have stayed 


the same 




the same 


□ (3) Training would have 


a 


(3) Training would have 


increased 




increased 



ADMINISTRATION 

1 9. Which statement best describes your relationship with your project monitor? 

□ 1 Helpful: Project monitor provided valuable advice and behaved in 
a professional and helpful manner. 

□ 2 Neutral: Projectmonitor was neither particularly helpful, nor 
difficult to work with. 

□ 3 Unhelpful: Project monitor, was difficult to work with did not 
provide helpful service. 

(If unhelpful, ASK PARTICIPANT TO ELABORATE): 



20. Which statement best describes your experience completing the procedures required 
to keep track of SOST hours? 

□ 1 Procedures were clear and easy to complete 

□ 2 Procedures were initially difficult but we mastered them over time. 

□ 3 Procedures were confusing and difficult and were a continuing problem 

throughout the project. 

(If #3, PROBE FOR REASONS): 
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21. Did you use a consultant to help you with the administrative process? 

□ (1) Yes 

□ (2) No 
IfYes, (ASK): 

How would you rate the performance of your consultant? 

□ Excellent 

□ Good 

□ Average 

□ Fair 

□ Poor 
If Fair or Poor, 

(ASK): 

Why? 



O 
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Appendix E 

SOST Interview Guide ETP Executives 



Date: 

Interviewee: 

Interviewer: 



1. How have SOST policies evolved over the last three years? 

• What factors have driven the policy changes? 

• How do you see the Panel's current interest in SOST? 

• What are the current issues related to SOST? 

2. What would make an ideal SOST component? 

• What problems have you seen? What has gone wrong? 

3. What role do you see SOST currently playing in the delivery of ETP projects? What 
role should it play? 

4. How would you evaluate the current reimbursement rate of $8.00 per trainee hour? 

• Is it still adequate? 

• What factors should be considered when setting the price? 

5. Which constituent groups are most interested in SOST? 

• What are each groups interests in the issues 

6. Is there anything else you want to tell us about SOST that we have not mentioned? 

7. Would you consider eliminating SOST from the ETP program? 
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Appendix F 

SOST Interview Guide ETP Monitors 



Date:^ 

Proj ect: 

Interviewer:^ 

Interviewee : 

Office:^ 

Describe purpose of the project. (We are evaluating the SOST component of the ETP 
project. We are not here to evaluate your particular project. We just want to see how 
SOST is being implemented) Assure the interviewee about confidentiality. 

(NOTE: (Get copies of recent monitoring reports) 

1. Before we begin talking about this project, I would just like to get your views 
on SOST in general. What particular problems have you encountered 
monitoring SOST in projects? 

2. In your view what features does effective SOST have? 

Lets talk about the Project now. 

3. Who is the person on the site primarily responsible for implementing the ETP 
project and the SOST component in particular? 

• Is the person a consultant or a company employee? 

• What is their best contact information? 

4. Have you observed any SOST at the site? (If YES) Could you describe what 
you have observed about how SOST is being implemented in this project? 

• Who are the instructors? 

• How were they trained? 

• How is the training actually delivered? 

• Is there a structure? 

• Are there planned activities and exercises or is SOST on an "as needed 
basis"? 

• How is mastery of the various competencies measured? 
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5. How do you think SOST could be improved? 

6. What do you see as the real costs of SOST and how do those costs relate to 
the current reimbursement rate? 

7. Should SOST continue? If yes, why? If no, why not? 



Thank you for your help! 
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Appendix G 

SOST Trainer Interview Guide 



Date: 

Project: 

Interviewer: 

Interviewee: 

Affiliation: 

Company Employee Title 

Consultant Company 

Describe purpose of the project. (We are evaluating the SOST component of the ETP 
project. We are not here to evaluate your particular project. We just want to see how 
SOST is being implemented) Assure the interviewee about confidentiality. 

1 . How did you get involved in this ETP program? 

• Were you involved in planning the project? 

• Did you plan the SOST component? 

2. What other experience(s) have you had as a trainer? 

• Have you done other types of OJT training or classroom training? 

3. Did you receive any specific training to be an SOST trainer? Please describe. 

• How would you evaluate that training? 

4. In a typical week, would you describe how you deliver SOST? 

• Are there structured assignments? 

• Do you have a regular schedule for seeing each trainee or is training mostly on- 
demand and/or troubleshooting? 

• Are you still responsible for your routine duties while you are delivering SOST? 

• How do you know if trainees have mastered the required competencies? 

• Could you describe a typical day or other time period when you are delivering 
SOST? 

5. What record keeping do you do to keep track of SOST? 

6. Overall, how effective would you say SOST is? 

e How does its effectiveness compare to classroom training? 

• What factors influence SOST's effectiveness? 

9 What changes would you recommend to improve SOST? 
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7. How are you compensated for being a SOST trainer? 

• Regular pay 

• Bonus 

• Other expenses reimbursed 



A couple questions about you and we are done. 

What is your job title? 

How long have you worked with the company? 
What is your hourly wage? $ . 



Thank you for your help! 
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Appendix H 

SOST Trainer Shadow Protocol 

Employer: ^ Date of Visit: 

T rainer : 

1 . How long are the interactions? A related question is how is time spent, if there are 
interludes where no instruction is going on, record them. 

2. How many participants in each interaction, is it one-on-one? A pair? A small 
group? 

3. What happens in the interaction? What topics are discussed? What activities take 
place? What does the instructor do? What do the trainees do? 

4. What is the outcome of the interaction? Do trainees attempt a new skill, change 
something about their work method, etc? Do they seem to learn an)dhing? 

5. Your own reaction, what do you think is really going on? Was this effective or 
ineffective interaction? What questions does it raise? 

SOST Observation 

Site: Date: 

Observer: 



Time 


Observation 

1. Participants 

2. Topic 

3. Activity 

4. Outcome 


Notes 
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Appendix I 

Trainees Questionnaire 

1 . What is your current job title? 

2. How many years have you worked for the company years 

3. How often do you get coaching from the SOST instructor? (CHECK ONE BOX 
ONLY) 

□ (6) Every day 

□ (5) 3-4 times per week 

□ (4) 1-2 times per week 

□ (3) Less than once a week 

□ (2) Varies from week to week 

□ (1) Never 

4. Typically, when your instructor does SOST with you, how much time does he/she 
spend? 

□ (1) Less than 15 minutes 

□ (2) Fifteen to 30 minutes 

□ (3) Forty-five to 60 minutes 

□ (4) More than 60 minutes 

□ (5) Amount of time varies from session to session 

5. Which of the following methods of SOST delivery best describes the SOST you 
receive? (CHECK ALL THAT APPLY) 

□ (1) I have structured exercises or practice I must complete on the job. 

□ (2) We slow down or stop production to practice skills. 

□ (3) When I have a problem, the SOST instructor spends time with me to 
solve it. 

□ (4) None of the above 

□ (5) Other — Please describe: 



6. Think about your experience in the classroom part of training and in the SOST 
part of training and then please rate the quality of the following on a five point 
scale where 1 -poor and 5=excellent: (CIRCLE ONE NUMBER FOR EACH) 





Class/Lab 


SOST 




Poor 


Avg 


Excellent 


Poor 


Avg Excellent 


Instructor 


1 2 


3 


4 5 


1 2 


3 4 5 


Overall Quality of 
Training 


1 2 


3 


4 5 


1 2 


3 4 5 
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7. How does the SOST instructor know if you have learned the skills or completed 
the assignments? (CHECK ALL THAT APPLY) 

□ ( 1 ) I tell him or her. 

□ (2) He or she observes me 

□ (3) He or she reviews my work products 

□ (4) I complete a test. 

□ (5) Other: 

8. The amount of SOST time compared to classroom time is: (CHECK ALL THAT 
APPLY) 

□ (1) Too Little 

□ (2) About Right 

□ (3) Too Much 

□ (4) I did not take classroom instruction 

9. How effective were classroom/lab instmctors and SOST instructors in teaching 
you the new skills? (CIRCLE THE APPROPRIATE NUMBER) 





Not Effective 
At All 














Very 

Effective 


Class/lab 

Instructors 


1 2 3 


4 


5 


6 


7 


8 


9 


10 


SOST 

Instmctors 


1 2 3 


4 


5 


6 


7 


8 


9 


10 



10. How useful were classroom/lab and SOST in enabling you to apply the skills you 
learned? (CIRCLE THE APPROPRIATE NUMBER) 





Not Useful 
At All 














Very 

Useful 


Class/lab 

Instmction 


1 2 3 


4 


5 


6 


7 


8 


9 


10 


SOST 


1 2 3 


4 


5 


6 


7 


8 


9 


10 
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11. How much impact have the skills you learned in classroom/lab and SOST had on 
your productivity? (CIRCLE THE APPROPRAITE NUMBER) 





No 

Impact 














Positive 

Impact 


Class/lab 

Instruction 


1 


2 


3 


4 


5 


6 


7 


8 


9 10 


SOST 


1 


2 


3 


4 


5 


6 


7 


8 


9 10 



12. Overall, how valuable would you say SOST was in improving your skills? 
(PLEASE CIRCLE THE APPROPRIATE NUMBER) 



Not Valuable 
at All 














Extremely 

Valuable 


1 2 


3 


4 


5 


6 


7 


8 


9 10 



Thank you very much for your help! 
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